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Preface 


For the thirty-fourth year, the Research and Theory Division of the Association for Educational 
Communications and Technology (AECT) is sponsoring the publication of these Proceedings. Papers 
published in this volume were presented at the annual AECT Convention in Anaheim, CA. A limited 
quantity of these Proceedings were printed and sold in both hardcopy and electronic versions. Volumes 1 
and 2 are available through the Educational Resources Clearinghouse (ERIC) System. Proceedings volumes 
are available to members at AECT.ORG. 

The Proceedings of AECT’s Convention are published in two volumes. Volume #1 contains papers dealing 
primarily with research and development topics. Papers dealing with the practice of instructional 
technology including instruction and training issues are contained in Volume #2. This year, both volumes 
are included in one document. 

REFEREEING PROCESS: Papers selected for presentation at the AECT Convention and included in these 
Proceedings were subjected to a reviewing process. All references to authorship were removed from 
proposals before they were submitted to referees for review. Approximately sixty percent of the 
manuscripts submitted for consideration were selected for presentation at the convention and for 
publication in these Proceedings. The papers contained in this document represent some of the most current 
thinking in educational communications and technology. 


Michael R. Simonson 
Editor 
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Introduction 

With increases in technological advances and Internet accessibility, participation in research and 
collaboration activities within higher education institutions are allowing students to engage in international learning 
and development opportunities within an online environment. Moreover, exposing students to international 
collaboration activities with others from a geographically distant location (Oncu & Cakir, 2011; Spector, Merrill, 

Van Merrienboer, & Driscoll, 2007) with culturally diverse participants, prepares them for the global workplace. 
Fortunately, the Internet and social network sites allow for the ease of collaboration globally. Not only is the 
international opportunity beneficial for students, but also is valuable for faculty research in designing Twenty-First 
century learning environments. According to one U.S. based survey report, 78% of faculty and 87% of students 
perceived online learning as essential for the Twenty-First century learning environments (CDW-G, 2011). 

Moreover, 64% of online students in another survey reported use of online collaborative applications several times 
monthly to connect with classmates on assignments (Koh & Lim, 2012). Thus, using technology through social 
media sites and collaborating internationally is essential to incorporating into higher education classrooms. 

This paper presents information on the first phase of laying the groundwork for best practices of online 
collaboration among students and two professors from two different cultures — Indonesia and the United States. We 
share the initial results of the collaborative learning experience in an online research project between Indonesian and 
American students, and explored research and future implications of cross-cultural collaboration in online learning 
environments. 

Context and Participants 

In the Spring 2013 semester, two international business classes collaborated on an international research 
project whereby each class conducted research and shared the results of the research between both classes. One class 
was a senior-level online international management class at a U.S. Southeastern university and the other class was a 
freshman-level international economics campus class from an Indonesian university. Both classes collaborated in 
both an online synchronous live question and answer session using Adobe Connect and an asynchronous discussion 
thread forum using a Google+ Community in order to learn about each country’s culture and global business 
practices. 

An important goal in teaching international business is to engage students through collaborative research 
projects so that they learn about other cultures and ways of doing business globally. Additionally, with increases in 
technological advances, asynchronous and synchronous online collaboration is possible across the globe now more 
than ever. Therefore, it makes sense for an international business class to utilize the technologies that allow for 
cross-cultural informational exchanges in an online environment (Edwards, Crosling, Petrovic-Lazarovic, & O’Neill, 
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2003). For two universities — one in Indonesia and the other in the United States — the online collaborative research 
project is a great opportunity for two cultures to learn about each other and exchange ideas on important global 
business practices. 

Southeast Asia is continually expanding as a big player in global economics and international trade. 
Furthermore, Indonesia is a developing Southeast Asian country that is continuing to grow economically and is 
becoming a key location for many manufacturing industries and exporter of resources (McFarlin & Sweeney, 2012). 
For Indonesian business students, understanding U.S. business practices is relevant and important for learning about 
international opportunity and trade. For the U.S. business students, learning about and understanding Indonesia and 
Southeast Asia is critical for future business partnerships. Thus, collaboration between an Indonesian and American 
university for two international business classes made sense to explore. 

Preparation 

By being introduced through a mutual academic colleague, a professor at a university in Indonesia and a 
professor at a U.S. Southeast university connected and began planning for a Spring 2013 research project between 
the international business classes, in September 2012. The plan was for each of the classes to research in areas of 
green initiatives and sustainability, human resources practices, small business practices, financial markets, supply 
chain management, technology and innovation, and corporate social responsibility. A total of 55 students were 
involved and each class was split into groups of four with each group researching a specific topic. There were a total 
of 14 groups. Moreover, each group with a specific topic was partnered with the other university’s corresponding 
group with the same topic. The groups researched their specified topic, wrote a paper, and presented an audio 
presentation in an online forum within Google + Community. In this online forum, students participated in 
discussion threads that included getting to know one another and collaborating on each group’s research. After three 
weeks of asynchronous online collaboration through discussion threads, a one-time synchronous live online forum in 
Adobe Connect allowed for students to ask questions and gather information about each group’s research. This is a 
first-time online international collaborative experience for both of the professors and the business colleges. Needless 
to say, both professors were very excited to venture out on this educational expedition. 

The Indonesian professor and the U.S. professor collaborated extensively in the planning of the project 
between the classes. At the conclusion of the semester, the professors had approximately 17 emails each between 
them, collaborated in seven Skype sessions of approximately 45 minutes each, and had met in person about four 
times during the U.S. professor’s trip to Indonesia in January 2013. Extensive preparatory collaboration and getting 
to know the other persons involved is key to planning a successful international online research project between two 
classes. Additionally, when in Indonesia, the U.S. professor visited the university by touring both campus sites, had 
dinner and breakfast with the dean of the business school and the professor of the Indonesian business class, and had 
a campus visit at the Indonesian business school that included both U.S. and Indonesian faculty, staff, and 
administrators. 

The Collaboration Process 

At the beginning of the semester, the two classes were required to post an introduction of themselves in the 
Google+ Community along with pictures representing some aspect about themselves and their city or country of 
residence. The Google+ Community was the chosen site for the international collaboration between the two 
universities. Some of the student information required for posting includes: 

1) University and anticipated degree; 

2) major; 

3) anticipated graduation date; 

4) current employment or campus activity involvement; 

5) international travels and locations; 

6) personal interests, passions, and hobbies; 

7) one unique or interesting fact about themselves; and 

8) industry or job position desired after graduation. 

The Google+ Community was also the location for instructions of the collaboration project including, the 
group audio presentation directions, the discussion thread area for the paired group’s posted research questions to 
the other group’s audio presentation, and the discussion area where the audio presentation link from either YouTube 
or Screencast-o-matic was posted for the whole community to view. In addition to these assignments, each class was 
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to write a group research paper on the specified topic. The students submitted the research papers as a separate 
assignment to their respective professors, but did not share the papers among the other students. However, the 
groups created audio recorded presentations using Microsoft PowerPoint and Screencast-o-matic. Audio 
presentations assignments were then posted in the Google+ Community and shared for both classes to view. The 
students were allowed to view all of the 14 group audio presentations for a total of one week. However, student 
groups were assigned to a partnered group from the other university with the same specific topic, and were directed 
to focus questions and interactions between the paired groups in the Google+ Community. Each group created and 
posted two questions for the other university group’s audio presentation, regarding their research or country. The 
questions were to be reflective of the groups understanding of presentation information or were to be questions 
asked regarding business practices or specific country information. In the Spring 2013 first collaboration project 
experience, the professors decided to have only the paired groups interact; however, the professors will redesign 
future collaboration projects to allow all students to interact with all groups. 

Close to the end of the semester, the live online meeting collaboration in Adobe Connect was set for all 
students and professors to attend with all attendees viewing and participating in the collaboration at one time. Since 
there was a twelve -hour difference, it was decided that the Indonesian class would meet in the morning during their 
regular class time of 9:00 a.m., while the U.S. online class met at 9:00 p.m. at night viewing from their own 
computers at home. The Indonesian class meeting was set up with a projector screen to view the Adobe Connect 
meeting. Additionally, about three laptop computers were used by the Indonesian groups for the interaction with the 
American groups during their designated group meeting time. 

Prior to the Adobe Connect meeting, both classes were given an agenda of which groups would go first in 
their question and answer session. The meeting was successful and lasted close to two hours. However, there were 
broadband issues early during the meeting and due to the heavy Internet use at a busy time of the day in Indonesia, 
the Adobe Connect meeting would only allow a few students’ webcams at a time. Thus, the student groups were set 
for the question and answer session by specified time, and were the only students allowed to interact at one time. 
Other students were allowed to view, and waited for their group’s question and answer time slot. Additionally, it 
was decided that no microphones would be used since there was tremendous feedback and noise at the beginning of 
the meeting. Although not ideal, the students resorted to the chat room feature within Adobe Connect to ask and 
answer questions. Although faces could be seen by webcam, the microphone feature was not used, except for the 
U.S. and Indonesian professors as co-facilitators. 

At the conclusion of the collaboration project, both professors required group peer evaluations and 
completion of a questionnaire about the collaborative research project. Since both professors want to continue the 
collaboration effort for future semesters, refining and improving both the process and the content of the project is 
important for both professors and for the success of the university relationships. Additionally, using a questionnaire 
instrument is key in both future process improvement initiatives and for desired collaborative research between the 
two professors. 

The Survey 

At the conclusion of the collaboration research project, a twenty-eight-item survey attempted to capture the 
student perception of the overall international collaboration experience. Some of the survey items included: 

1) basic demographic information; 

2) uses of the Internet; 

3) perception of clearly articulated instructions on collaboration requirements 

4) perceived gained knowledge of foreign business practices after completion of the international collaboration 
project; 

5) perceived gained knowledge of the other culture after completion of the international collaboration project; 
and 

6) suggestions for improvement for future international collaboration projects. 

Of the 55 students surveyed, over 90% of the students reported that they both enjoyed and learned from the 
international collaboration experience. Additionally, many students stated that they would welcome the opportunity 
again to collaborate internationally. However, some of the feedback on dissatisfaction with the experience included: 

1) webcam and sound technical difficulties when meeting in Adobe Connect; 

2) time difference in meeting late at night; and 


339 



3) age difference and maturity level of students. 

To address the dissatisfaction feedback, both professors discussed the need for additional orientation for the 
students at the beginning of the semester (Angelino, Williams, & Natvig, 2007) and for changes in meeting times 
and protocol in Adobe Connect to address the technical challenges. Additionally, etiquette protocol will be added so 
that there is less possibility for misinterpretation of non-verbal behavior during the meetings. 

Lessons Learned 

While the first international collaboration experience with both universities using social media tools for 
collaboration was an overall positive experience for all involved, it was not without its challenges and learned 
lessons. Since both professors will continue their relationship and future collaboration efforts, refining the project is 
very important. Furthermore, both professors will continue to explore other research on international collaboration 
and will continue collaborative meetings through Skype. Following is a brief checklist of changes and 
considerations for the following semester project collaboration efforts: 

1) The meeting time for Indonesian students will be in the evening rather than in the morning when Internet use 

is less; 

2) deadlines will be set for posting in the discussion forum for both presentation URL link and questions rather 
than allowing one week to post; 

3) a trial run meeting will be set in Adobe Connect prior to the actual scheduled meetings; 

4) early ten minute minimum meeting times will be required prior to Adobe Connect meetings to set up 
technical devices and meeting rooms; 

5) multiple meetings will be set in Adobe Connect over a few days for fewer groups and thus, less people at one 
time in the Adobe Connect meetings; 

6) students will be required to use built-in headset microphones to reduce noise interference; 

7) two Adobe Connect breakout rooms will be set up to allow four groups to collaborate at once; 

8) added instructions will be displayed in the main meeting room of Adobe Connect; 

9) email reminders to all students will be sent twice, prior to Adobe Connect meetings; 

10) a reflection assignment will be added for students to give detailed feedback on what they have learned from 
the experience; 

11) other strategies including participation points will be implemented to encourage more collaboration among 
the students in addition to their paired groups’ research; 

12) additional strategies will be included to encourage other group interactions; and 

13) additional strategies will be implemented to encourage other creative posts in addition to the discussion 
thread introduction assignment that may include self-made student videos to further the cultural experience. 

Cultural Considerations 

It is important to understand the culture that one works with as much as possible by reading cultural 
materials and by getting to know the people involved in the collaborative project. The Indonesian professor was 
helpful in filling in the expectations and protocol of the Indonesian students. However, according to Hofstede’s 
Cultural Dimensions research, Indonesians have a high regard for structure, direction, feedback, and leadership from 
those in charge — the professors (Bell & Zaitseva, 2005; Hofstede, G., 2001). Conversely, American students tend to 
venture on their own with verbal expression regardless of power structures (Bell & Zaitseva, 2005). Knowing these 
tendencies, it is important to encourage respectful and frequent collaborative discussion from both the Indonesian 
and American students and for both instructors to provide clear guidelines and feedback in encouraging 
collaboration. While both professors know that there will be cross-cultural differences in ways of expression and 
communication between Indonesian and American students, the collaboration experience will richly enhance both 
cultures’ understanding and knowledge of the others’ (Kim & Bonk, 2006). 

Conclusion 

The importance of incorporating international collaborative projects using social media sites in higher 
education is key to preparing today’s students for Twenty-First century global workplace skills. Both professors see 
the value in bringing to the classroom an international perspective that cannot be obtained through a textbook. 
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Additionally, introducing global business practices and building international relationships is key preparing students 
for today’s global workplace. 
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Online gaming is no longer just a leisurely activity. Over the last five years, education has begun 
repurposing many online games and virtual world environments to actively engage students with a new delivery 
method to accomplish learning goals and objectives. Unequal representation of ethnicity in gaming puts minority 
players at a disadvantage in terms of making a psychological connection with their virtual self thereby greatly 
diminishing game play. 

The purpose of this research is to explore the options available to users to represent themselves in terms of 
skin, eye, and hair color. The default options one chooses in an attempt to get as close as possible to one’s own 
representation will also be investigated. If a bias does exist in gaming platforms that are highly targeted by 
educators, then it is important that the gaming community is made aware of the impact it has on minority users and 
their gaming experience and reasonable solutions are offered. 

The use of avatars in gaming extends the bounds of immersion to a level of realistic representation. When 
one sees himself or herself as a character interacting with the environment using chat features, movement, and object 
manipulation, the experience becomes much more personal. A psychological relationship develops between the user 
and the avatar (McCreery, Kathleen, Schrader, & Boone, 2012). The player uses this virtual identity for months, 
sometimes years existing through the eyes of this virtual body. The avatar becomes a virtual extension of the user. 

The foundation of this relationship begins with choosing the characteristics. Most online gaming offers 
users the opportunity to choose a name and a physical representation of themselves using a series of features. When 
users are choosing humanoid characters, they often have the ability to choose skin, hair, and eye color. Options to 
change the mouth, nose, and body shape may also be available. The characteristics that one chooses culminate into a 
digital rendering of themselves into the virtual world (McCreery, Kathleen, Schrader, & Boone, 2012). The visual 
image not only defines a part of the character but also how it will be viewed and interacted with by others. 

Research into minority representation in online gaming environment is scarce. In a study conducted by 
Williams, Martins, Consalvo, and Ivory (2009) the authors found “a systematic over-representation of males, whites, 
and adults” (p. 815). Tanner (2009) concluded that online games such as the World of Warcraft and EverQuest 
default selection for ethnic representation are white while alterations to skin is an “exotic deviation” (p. 3). Further 
supporting evidence of this is in MapleStory, where users must use the game’s currency called Karma Koins that has 
to be purchased with real U.S. currency in order to change their skin color to darker shades. 

Kafai, Cook, and Fields (2010) conducted a study of an interactive role playing virtual world called 
Whyville. The authors found that there was a lack of darker skin feature selections. Players that did attempt to 
change their face to a darker color could not find matching body parts when changing other things about themselves 
like their clothing that only came with light skin features for the attached arms or legs. These “two toned” players 
were made fun of and some were subject to racial jokes and slurs. Being that physical avatar representation has a 
psychological connection to how players identify with themselves in the game and its interactions, the availability of 
adequate selection of multiple shades of darker pigmented skin is essential to the games immersive quality. 

Online gaming is a valuable tool for educators and instructional designers to actively engage students. 

When applied in a meaningful way, it can harness the power and creativity of students’ minds, increase knowledge 
transfer, and cement the learning objectives and goals through the learning experience. The social and psychological 
aspects of this environment cannot afford to be ignored by educators since many lessons taught in the traditional and 
virtual classroom are an extension of life itself. Educators and instructional designers must be mindful of this, 
especially when working with K-12 students who often go through periods of identity searching and sometimes 
crises that can adversely affect learning. Without proper minority selection choices, the gaming community is forced 
to ask itself, what is it saying about the importance of different groups of players’ experience? 

The internet provides a space for Earth’s ethnically rich and diverse users access connect across the barriers 
of geography, language, and culture to meet in online games. When asking someone to represent him or herself in a 
game, it is an attempt to make a connection with the user and create an enhanced gaming experience. Maximizing 
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this connection by having an adequate amount of characters or traits supports the goals of using an avatar and 
creating the best possible experience for the user. 
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Abstract 

The school technology specialist and media coordinator are sometimes perceived as support personnel 
without the status of instructional leader in the school. By applying common characteristics for the teacher leader, 
servant leader, and the job functions of school technology specialists, there is opportunity to transform the image 
from support personnel to instructional leader and change agent. One strategy examined for this study was the 
integration of community service projects embedded in internship projects for master of education students. Interns 
were able to apply their expertise as instructional technology specialists while serving students, teachers, and the 
community through special service projects uniquely designed for their region of the state. 

Teacher Leaders. 

Current research in K12 education reports an increasing emphasis on the importance of teachers as leaders 
in their school. Recent reports expand the role of school leader to include community leader as well. Trae Stewart 
(2012) brings to our attention the importance of teachers as transformational leaders who influence students, parents, 
and their peers. A good leader might be called upon to “model meaning in life” (Stewart, p. 233). Leaders impact 
school policy, design of curriculum, and climate in the classroom. Harrison and Killion suggest that characteristics 
of teachers as leaders are situational and can vary based on setting and community. Leadership opportunities might 
occur in one or more situations in a school — resource providers, instructional or curriculum specialist, classroom 
supporter, learning facilitator, mentor, school-wide leader, data coach, catalyst for change, and learner. 

One of the most important areas for leadership is the potential for creating vision and guiding others in 
professional growth (Daft, 2005). Learning Forward, the Professional Learning Association serves as an advocate 
for strong teacher-leaders in K12 schools. The mission of Learning Forward is to provide professional development 
for practitioners as well as report current research in best practices for leadership and professional learning in 
schools. Standards for Professional Learning have strong emphasis on leadership. “Professional learning that 
increases educator effectiveness and results for all students requires skillful leaders who develop capacity, advocate, 
and create support systems for professional learning”, (Learningforward.org, online). 

Technology Specialists as Leaders. 

Teachers identified as professional development leaders or instructional technology specialists have 
opportunity to become visionary leaders (Crippen, 2005; Darling-Hammond, LaPointe, Meyerson, Orr, & Cohen, 
2007). Collaborators in the International Society for Technology in Education have developed standards for 
“Advanced Digital Age Coaching” (ISTE, 2012). Standards for leaders were most recently developed and added to 
the original NETS standards for students and teachers. The first heading. Visionary Leadership, includes four areas 
of competency, all of which could be used to guide, mentor, and coach other teachers as they grow professionally, 
especially in the use of digital resources. Association for Educational Communication & Technology professional 
standards for programs in school library media and educational technology specialist programs promote leadership 
skills through Standard 4. Candidates in a media or technology program of studies must, “Demonstrate leadership 
attributes with individuals and groups...” (Earle & Persichitte, 2000, p. 26). The technology specialists is in an ideal 
setting for providing vision, guiding, mentoring, and promoting professional growth for teachers. However, many 
technology specialists experience negative responses from teachers when seeking collaborative projects. Others 
report frustration when enticing teachers to attend and engage in professional development experiences. Could the 
characteristics of a servant leader provide a solution for these concerns? 
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Servant Leaders. 

The term “servant leader” became known through the work of entrepreneur Robert Greenleaf (1977, 2002) 
According to Greenleaf, a servant leader has impact on an organization through thoughtful guidance designed to 
promote the welfare of employees and other administrative personnel. Greenleaf admitted the term servant leader is 
an oxymoron but brought the two contradictory terms together as a strategy for changing conceptions of 
organizational structure. Based on personal experience in corporate world, and anecdotal evidence, Greenleaf 
believed the best way to bring about change is by modeling leadership through service to others. Larry Spears 
followed the work of Greenleaf using a more scholarly approach. Spears proposed ten characteristics of a servant 
leader (Spears, 1998, 2010). An effective leader should demonstrate skills in listening, empathy, healing, awareness, 
persuasion, conceptualization, foresight, stewardship, commitment to growth in people, and building community. 
Using these characteristics, many schools and teachers are becoming advocates for community service (PSEA, 

2012; Colby, Bercaw, Clark, & Galiardi, 2009). 

Leadership Through Internship Experiences. 

Professional programs typically include an internship or practicum. The purpose is to provide practical 
experiences within monitored safe environments. Interns are able to apply theory to best practice without the 
pressure of actual on-the-job risks. Darling-Hammond (2007) and her colleagues have reported case studies in the 
area of leadership, primarily in K12 settings. Based on her findings, one could expect experiential learning and 
simulations to provide the best programs for teaching leadership skills. Internships make good use of the workplace 
as a site for leadership planning that enable candidates to apply leadership knowledge and skills under the guidance 
of an expert practitioner. A well-planned internship experience will include opportunities to apply theory to best 
practice with regard to the profession but also through opportunities for leadership. 

Finding Common Characteristics. 

A review of the literature was used to identify and analyze attributes for various characteristics associated 
with leadership. Harrison and Killion (2012) describe the following as characteristics of a teacher-leader. 

Teacher-Leader Characteristics. 

• Resource Provider. Teachers who help their colleagues by sharing resources are leaders through positive impact 
resulting from a collegial attitude in the workplace. Shared resources might include books, teaching aides, 
websites, and instructional software. 

• Instructional or Curriculum Specialist. Teachers who help their colleagues implement effective teaching 
strategies will be looked to as instructional leaders and will often become the go-to person in a school. Teachers 
who have deep understanding in how curriculum can be planned to support assessment and student learning 
outcomes will gain reputation as an important leader in their school. 

• Classroom Supporter. Harrison and Killion describe this person as one who helps other teachers implement new 
ideas through coaching, modeling, and demonstration. 

• Learning Facilitator. This teacher shows leadership skills through planning of professional development and 
growth for others. They often serve or lead committees related to staff development and curriculum reform. 

• Mentor. This teacher comes alongside others who are novices or need the support of another professional. 

• School Leader. This teacher has keen interest in serving and leading committees to bring about change or 
improvements in the school. 

• Data Coach. This teacher is a visionary who can transform data into best practices in teaching and instructional 
methods. 

• Learner. Harrison and Killion consider this an important attribute. Leaders must adopt the value of life-long 
learner. 

Servant Leader Characteristics. 

According to Spears (2010), the following ten characteristics are found in servant leaders. Compare these 
characteristics with those listed above for teacher-leader. 

Listening. A servant leader has mastered the skill for listening to provide appropriate and useful feedback to a peer 
or employee. 

• Empathy. Provides appropriate concern and understanding in the needs and unique value of others. 

• Healing. This characteristic is associated with promoting restoration of relationships in the workplace. 
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• Awareness. Having knowledge and understanding in the ethics, power, and values of the environment. 

• Persuasion. The servant leader has special wisdom in persuading others to strive for goals that meet the greater 
good. 

• Conceptualization or Foresight. This person is able to reflect and self-assess his or her own strengths, abilities 
and apply these to a vision for the greater good. 

• Stewardship. The servant leader is able to apply the use of resources in appropriate and timely ways. 

• Commitment to the Growth of People. According to Spears, this is one of the key attributes of a servant leader. 
The professional growth and well-being of others is paramount in his or her leadership style. 

• Building Community. The value of shared ideas, responsibilities, and goals remains in the forefront of his or her 
leadership activities. 


Research Questions 

North Carolina schools value the contributions of technology facilitators as integration specialists, staff 
developers, and peer coaches. The position of technology coordinator, requiring advanced school licensure, or 
technology facilitation (special endorsement required) is typically fdled with experienced teachers who have a 
proven track record as leaders and teachers in their schools. Most schools rely on the technology facilitator to serve 
as staff development liaison between central administrative offices and the building level administrators. The North 
Carolina Department of Public Instruction has a published job description for these positions. ( See 
http://www.ncpublicschools.org/docs/work4ncschools/employment/jobdescrip/techfacilitatorjob.pdf) Main functions 
are listed within three categories, one of which includes leadership responsibilities for working with teachers, 
administrators, and in the area of teachers’ professional development. A close analysis of the job functions for the 
technology facilitator, classroom teacher, and servant leader indicates there is overlap and commonalities across the 
three sets of standards. Because of the high valued placed on teacher-leaders and because the technology specialist is 
in a natural position to serve as leader, the author of this paper asked, how can candidates for the Master of 
Education in Instructional Technology with school licensure gain experiences as servant leaders and teacher leaders 
in their schools? Could a technology specialist apply his or her expertise in use of digital resources through service 
projects and partnerships between school and community? What impact would these projects have on the 
development of characteristics for servant teacher leadership in the school? Would the characteristics of servant 
leader have positive effect on future collaboration between technology specialist and teacher, and between schools 
and the community? 


Methodology 

Candidates in the MAEd in Instructional Technology must complete 110 hours internship in a school 
setting. Because the degree is 100% distance delivered, interns are located across the state in more than 100 different 
school systems. Regions within the state are mountainous, seacoast, urban, and rural. Because of the diversity of our 
students, internship proposals are as varied as the cultural characteristics and environment of our students. Over 
three semesters, qualitative and quantitative data was collected during the graduate course EDTC 6992 Internship 
Seminar. Students in the internship were assigned a community service project to be included in their internship 
proposal. The interns were to form a partnership with teachers in their school or a collaborative project between the 
school and a community agency. The interns were assigned readings on servant leadership and teacher leadership 
characteristics. Based on the attributes of servant leader, interns explored possible projects in their school and/or 
community. The project proposal was submitted using a logic model format listing resources, objectives, proposed 
artifacts, photos, and documents, and description of short-term and long-term outcomes. At the end of the term, 
interns submitted a formal report with a completed logic model. Artifacts and evidences were aligned with original 
objectives. Reflections were submitted with how well the project achieved proposed outcomes. Interns were also 
asked to participate in an IRB approved survey with questions about the experience, usefulness for their job as 
technology facilitator, and impact on their role as a leader in their school. Using Qualtrics software, the survey was 
developed by the researcher reflecting the ten teacher leader characteristics proposed by Harrison and Killion and 
the ten servant leader characteristics proposed by Larry Spears. Results of the survey were analyzed using Excel to 
determine mean scores as interns reported satisfaction with various leadership, instructional, and service projects 
included in the internship. The final projects were designed to include photos, descriptive reports, timelines, and 
reflections on the community service project. Two areas of emphasis were required for the report: 1) how servant- 
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teacher leader characteristics were applied to the internship, and 2) how skills and concepts in instructional 
technology were applied to the internship. 


Results 

Analyses of characteristics for the teacher leader compared with servant leader showed alignment in several 

areas. 


Table 1. Comparison of servant-leader and teacher-leader characteristics in K12 environments. 


Servant Leader 

Teacher leader 

Listening 

Mentor 

Empathy 

Classroom Supporter 

Healing 

—not identified in Harrison and Killion 

Awareness 

Catalyst for Change 

Persuasion 

Instructional specialist and resource provider 

Conceptualization and Foresight 

School Leader 

Stewardship 

Resource person and Data Coach 

Commitment to the Growth of people 

Learning Facilitator and Resource provider 

Building Community 

Curriculum specialist, School Leader, and Learner 


Reflections and discussion threads archived from course materials were analyzed for themes related to leadership 
characteristics. The table below provides selected comments from the intern as she reflects on experiences that 
match characteristics of servant leaders, teacher leaders, and technology facilitator job description. 


Table 2. Reflections on teacher leader characteristics during technology internships in K12 schools. 


Teacher Leader: 

Resource Provider/data coach 

Servant Leader: 
Stewardship 

School Technology 
Leader Function: 

Evaluate and select digital 
tools and resources-best 
practices. 

“Teachers should ALWAYS be 
researching new strategies and 
methods for teaching their students.” 

“I helped teachers with district 
benchmarks now delivered online. 
This helped reduced stress [in the 
school].” 

“Make sure I stay 
abreast. . .teachers turn to 
me.” 




Teacher Leader: 

Mentor; Learning Facilitator 

Servant Leader: 

Persuasion; Building Community 


“Help by creating templates for 
vocabulary and technology related 
problem -based learning such as 
webquests.” 

“I begin by using the new resource 
myself and show teachers how it is 
beneficial. 

I ask teachers to show how they are 
using new resources and how they 
‘worked the bugs’ out.” 



In addition to submitting reflections on internship their experiences, students were asked to respond to an 
online survey with questions related to leadership experiences. Results of the survey administered at the conclusion 
of the internship showed that interns value several of the characteristics associated with servant leadership. Using a 
sliding scale (0 to 100) the interns rated the value of certain activities typically occurring during an internship. In 
Figure 1, see questions and ratings provided by interns during spring 2012 (N=24). 


347 




Nurture communities of practice... 
Establish and build small... 
Be involved in decisions and... 
Make every effort to help people... 

1 hold the resources and... 
Aware of future consequences to... 
Engage in a realistic response to... 
Recognize self-awareness can be... 
Maintain a practice of self-... 
Am honest and truthful in all... 
Am constantly aware of ethics... 
Look for ways to help others find... 
Intentional about seeking healing... 
express empathy by accepting... 

] 
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Figure 1. Results of survey indicating the value interns place on 
activities representative of servant-leader characteristics. 


Items in the survey were reflective of qualities identified in servant leaders. The highest scores showed interns 
valued building communities, being involved in leadership decisions, helping people grow professionally, being 
good stewards of resources, exhibiting awareness of potential outcomes from decisions, engage in reflection and 
self-wareness, honesty/truthfulness and showing empathy to others. 

Community Service and the Technology Facilitator. 

The interns reported innovative projects designed to meet authentic needs in the community. The creative 
efforts at applying their expertise as technology specialists were clearly evident throughout the three cohorts used 
for this study. The following examples show how the interns applied empathy, a desire to enhance professional 
growth in others, listening skills, persuasive strategies for achieving goals, stewardship of resources, and a desire to 
build social and educational community. 

I am going to put together a Technology Expo night directly after the October PTA meeting, where parents 
and the community will be invited to come out to our school to attend technology classes of their choice. 
Conduct videography and desktop publishing workshops to create a program for people who attend the 
special Veterans’ Day event. The Veterans’ Day Tribute will be for the community during the week of 
October 29 through November 6. 

The school that I work at has partnered with the non-profit organization. Girls on the Run. The goal is to 
inspire young girls and teach them life skills through the process or training for and running a 5K race. 
With so many teachers serving as coaches, I saw a need for improved collaboration and communication 
methods. . . 

Last year, one of the teachers applied for, and received, a grant to plant a seasonal vegetable garden for the 
students and community to plant, care for, and harvest. . . Since not every student and teacher can 
participate in the actual planting, upkeep, and harvesting, I have decided to work with the third grade 
teacher who wrote the grant to develop a Wiki page that students, families, and the community can access 
to learn about planting, caring, and harvesting a nutritional garden. 
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Create a Boy Scouts of America website for the eight Packs and seven Troops located within Richmond 
County. The homepage will be created to “advertise” scouting opportunities offered within our 
community. The website will display all Richmond County Troops and Packs, its members, and leaders. 
Contact information will be provided for those interested in joining. 

Reading Rocks. Family Reading. Several activities are planned K-5 Parents and children. These include: 
Presentation video- This highlights student reading program and the importance of reading with your child. 
Book Scavenger Hunt Students are given a clue to a book to find then read with parent. Matching game 
Match book title with author cards. Take the Reading Challenge- Parents and students pledge to read 20 
minutes or more together each day. 

Family Science Night. Objectives: 1. build important skills: self-discipline, time management, project 
management, and written, verbal and visual communication skills; 2. to show the importance of patience 
and precision ; 3. to be able to give the children a setting to bond with each other and their families and to 
gain new friends. 


One project was very well received in a military community in the eastern region of the state. The intern’s 
awareness of the needs of military families and empathy she experienced for children of deployed parents led to a 
vision for the following wiki: http://militaryfamilyresources.pbworks.com/ 

See figure 2 below with a screenshot of the student’s homepage. She used systematic methods for planning 
content and design of her wiki. She also conducted usability tests before making it available to parents. Feedback 
from teachers, parents, and military personnel were used to refine and perfect the content of her resources. 


Student Resources 

as tv J megaivrebnd 1 year ago 


Militory Kids Connect 

Th.< it c great rosoircc for students of ell ages Offers support from pre-deployment to the parent's return. 


Operation We Are Here 

Directory of resources for Militory children and teens 


PBS Kids 

Coming home... Militory families cope with change 


Deployment Kids 

Keeping in touch, downloadable activities, school end home resources 


Comments (0) 


You dont have pesmeson to comment on the page. 


) lo Mil this wofkstacc. mwefl access . 

Moody have an account? LoptV 

Navigator 

• Hftary ramty Resources Home Page 

• Parent Resources 

• SdeBar 

• SUyng Corrected 
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do »*o options O 
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UMc to yo-jr <*vor*a wab seat or inytrwig ns*. 
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J> Parent Resources 
(dted by megarurtbnd 


Figure 2. Technology skills applied to a community service project by intern in an 
instructional technology degree program 


The student was able to apply a wide range of leadership skills as she interviewed teachers and students in her 
elementary school. She worked collaboratively with a military liaison to help with the design of her wiki. Response 
from school administrators was especially positive indicating a much needed morale boost during times of heavy 
military deployment. Her project is representative of service, leadership, and instructional technology. 


Conclusions and Recommendations 


There are many commonalities between the characteristics of a servant leader and a teacher leader in K12 
settings. Internship reports also show alignment between professional standards for technology facilitators and their 
counterparts — teacher-leaders. The technology specialist is in an ideal profession for serving teachers with both 
instructional and technology support. It would be even more important to establish one’s self as an instructional 
leader and catalyst for change in the school. Positive professional image and establishment of productive 
relationships within a learning community are two of the potential outcomes associated with a leadership style that 
applies Greenleaf s servant leader attributes. 
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Community service sites seem to provide an excellent venue for servant leader internship experiences. The 
community sees the teacher as a positive role model for both students and adults in the school. The technology 
facilitator or other specialist has opportunity to demonstrate empathy and concern for colleagues, students, and their 
families. A positive caring image could lead to instructional projects that permit modeling of best practices in use of 
technology. Internships centered on community service provide authentic problem solving and positive personal 
experiences despite the pressure of academic demands. These must be well planned using systematic methods such 
as use of a logic model template. Careful planning, appropriate use of resources, and predictive timelines will 
ensure a positive experience for the intern while he or she gains new insights and skills as a leader in the school. 
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Abstract 

Group work has been a popular pedagogical practice because of its potential to integrate collaboration and 
learning. However, there are pitfalls to group-work projects such as disproportionate workload, appropriate 
assessment, and intergroup drama. This paper looks at the pedagogical application of the collaborative video log as 
group work through an illustrative case study. The collaborative video log is a learning model designed to engage 
groups of students around a topic that promotes ownership, reflection, and academic and social outcomes. A case 
study was conducted on a collaborative video log which consisted of five graduate student vloggers who together 
produced 36 videos for a technology and education course. Analysis of observation and interview data showed the 
CVL has a unique design which lends itself to ownership, academic outcomes, and social outcomes. The most 
common themes of ownership included technological ownership, ownership of the CVL structure, and ownership of 
personal learning. Prominent academic outcomes were building a learning community, realizing potential, and 
developing technological literacy. The three themes present in social outcomes were presenting personal 
information, embracing differences, and expanding relationships beyond the classroom. Though experienced 
drawbacks were few, potential drawbacks were suggested by the participants. 

Introduction 

Group-work projects in formal education are a common pedagogical tool for learning. Advocates for this 
pedagogical practice highlight both the need to develop collaborative work skills for practical application in the 
workforce and the benefits of constructive learning processes (Alden, 2011; Cameron, Morgan, Williams, & 
Kostelecky, 2009; Isaac, 2012; Jensen, Mattheis, & Johnson, 2012; Scherling, 2011). However, the pedagogical 
design and application of group-work projects have presented challenges for teachers and their students and can 
unintentionally result in negative cognitive and affective consequences for students (Issacc, 2012; Neu, 2012). 

Problematic group work can result in interpersonal conflicts or drama; ineffective communications; poorly 
distributed workload and unequal participation by group members (Isaac, 2012; Scherling, 2011; Thompson & Ku, 
2006). Students may lack an understanding of collaborative learning process and lack appreciation for the 
connection between social learning and social task development (Cameron, Morgan, Williams and Kostelecky, 

2009; Morgan, Cameron and Williams, 2009). Teachers face the task of determining a fair and equitable way to 
grade group projects that reflects both the groups interaction and the individual’s participation, or lack thereof 
(Alden, 2011; Dommeyerm, 2012). The challenge for instructors is designing group work that promotes 
collaborative learning, is fair and equitable for students, and minimizes the potential for the unintended (negative) 
effects for students. 

The worlds of informal online communities and formal educational classrooms are converging (Jenkins, 
2006). The internet is increasingly being seen as a way for people to connect via virtual worlds (Gottschalk, 2010), 
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games (Steinkuehler, 2006), and various social media. YouTube which was launched in 2005, has piqued the 
interest of researchers as an online community. As Burgess and Green (2009) suggested, 

The architecture of YouTube does not overtly invite community-building, collaboration, or 
purposeful group work .... YouTube’s architecture and design invite[s] individual participation, 
rather than collaborative activity; any opportunities for collaboration have to be specially created 
by the YouTube community itself, or by special invitation from the company, (pp. 63-65) 

A YouTube community and many subcultures have developed, and YouTube has become an interactive website 
where users share content and information as well as communicate through videos, rating, and text responses 
(Chang & Lewis, 2011). Meek (2012) claimed YouTube has also allowed for a “shift from media to social media 
[that] requires us to rethink the spatial relations between communication and politics in everyday life” (p. 1430, 
italics added). 

There is a growing body of literature written on the educational potential of YouTube. However, “novel 
research terrain brings with it novel difficulties” (Baym & Markham, 2009, p. xiii). Methodological texts tend to 
have a dearth of suggestions for conducting research on online inquiry. A vlog, also known as a video log or video 
blog, is a digital space where a creator regularly publishes content though video form that focuses on a specific 
theme or topic such as pop culture, an academic area, political interest, or one’s personal life. Another key feature of 
the vlog is that it is achieved through publication which is usually publically available on the internet. There are 
four different types of vlogging videos according to Werner (2012): the confession video, the reaction video, the rant 
video, and the witness video. Smith (2012) wrote about the sharing of intimate moments through vlogs. She found 
that participants developed relationships with people online through video-based social media sites; these 
relationships were valued as much as friends and family in real-life. He claimed vlogs promote dialogue, relay 
emotion, spread information, and archive expression. 

Educational researchers have also looked at the pedagogical implications of utilizing YouTube and video 
creation in the classroom (Cayari, 2011; Hrastinski & Aghaee, 2011; Hung, 2011; Jackson & Wallin, 2009; Juhasz, 
2008; Kellner & Kim, 2010; Lin & Polnecki, 2009). Research on video logs or vlogging in educational settings has 
focused on the utilization of individuals’ video logs instead of collaborative efforts and has been limited to short 
duration studies with only a few videos created over a short period of time (Hung, 2011). 

The Collaborative Video Log 

A collaborative video log (CVL) consists of videos created by a small group of people. The model used in 
this study involved each person being responsible for uploading a video on their respective day of the week. For 
example, one person posted a weekly video on Monday, the next person posted a weekly video on Tuesday, and so 
on through Friday. The participants developed questions which they answered throughout the duration of the project 
(seven weeks) and dialogic interaction between videos was encouraged. 

The CVL exemplifies the use of YouTube as medium for video creators to join into participatory culture, a 
term developed by Burgess and Green (2008) to describe how posting online video can be used to socialize with 
others and interact with media. However, the CVL seems to be completely absent from the research literature in 
sociology and education. Kinder (2008) suggested there are four rationales for online videos: conference, 
exhibition, precursor, and research. Each of which is incorporated in the CVL format. Juhasz (2008) emphasized the 
importance of finding ways to utilize technology not as a novelty, but as an appropriate, education enhancing 
method. Online video logs can be utilized to foster dialogic learning which is popular amongst critical educators and 
theorists (Jackson & Wallin, 2009; Kellner & Kim, 2010). 

The purpose of this research is to examine, via an illustrative case study, the potential pedagogical use of a 
CVL. The case for this study was the CVL based on a collaborative framework developed by a group of graduate 
students who were enrolled in an educational policy studies course dealing with education and technology reform in 
the Fall of 2012. This descriptive study focused on the following two questions: 

1 . What are the structure of and processes utilized in a CVL? 

2. What are the perceived benefits and drawbacks of utilizing a CVL in an educational setting? 

Methods 

Data in this study were collected using qualitative methods under a constructivist framework. Qualitative 
methods are appropriate for studies that further our knowledge of phenomena, context, and people within a bounded 
case (Creswell, 2012; Stake 1995). Methods in this study were drawn from internet based research (Hine, 2009; 
Orgad, 2009) and self-research (Bullough & Pinnegar, 2001); methods were then selected through a collaborative 
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and consensus building process among the study’s participants and the researchers. The participants were the five 
members of the CVL (hereafter referred to by the day of the week on which they posted), the course instructor, and 
teaching assistant. The instructor’s and teaching assistant’s perspectives were used to triangulate data from the CVL 
participants and help mitigate bias among the participants. Individual one-on-one semi-structured interviews were 
conducted with each of the participants. The interview questions are available in Appendix A and data collection 
and processing assignments can be found in Appendix B. The researchers observed the 35 publically available 
videos created by individual group participants and a final culmination video which featured all five members of the 
CVL group reflecting on their project via the group’s public YouTube channel 

( http://www.voutube.com/user/epsfourfifteen ). Observations were recorded using a priori and emergent categories 
that were developed among the researchers. The observation data and the interview data were then coded and 
recoded using a priori and emergent categories, until reoccurring themes could be identified and summaries written. 
This paper presents the most prevalent themes identified in the group members’ interviews, which were supported 
by the instructor and teaching assistant interview and the observations. 

The researchers in this study were also participants in the CVL. To mitigate biases, ideas presented in this 
paper were discussed extensively among the researchers until consensus was met. The remaining members of the 
group, who are not listed as researchers, provided member checks, authenticating that the views expressed in this 
paper reflect their perceptions of their experiences in the CVL. Despite our efforts to mitigate biases, it should be 
noted the involvement of the researchers in this case study as well as the relationships among the participants may 
limit participants’ willingness to share information they feel may be perceived negatively. The transferability of this 
study rests on the similarity of the scope and flexibility of the curriculum. As such, while we encourage educators to 
utilize CVLs, we encourage instructors to customize the project to best suit the needs of their program and students. 
Our hope is this research will provide educators with the framework under which they can explore applying this type 
of group project in their pedagogical practices in their educational setting. 

Findings 


The Structure of a CVL 

“The collaborative vlog project was based upon five different people with, in our case, very different 
backgrounds, picking a day during the week in which they would post a video on a given topic” (Tuesday). This 
description of a CVL provides a concise and accurate description of the structure of the CVL. This structure created 
what the participants described as a setting where meaningful interactions were happening asynchronously through 
technology. Participants described the CVL as a “conversation” or a “really long interview in small, digestible 
chunks.” The professor for the course described it as a set of “five video journals or diaries, of people who were 
using this format to think about and reflect on various things that were going on in the class as the class was going 
along.” 

Early in the CVL project the group met face-to-face and defined the scope of the project. This meeting was 
described as similar to those that take place in traditional face-to-face groups. This meeting served two purposes. It 
was crucial that each member understood the project and what their role would be throughout the process. This 
shared understanding formed a foundation upon which the group was able to collaborate and avoid the pitfalls which 
are common with group work. One member of the group related, “I honestly had no idea what a collaborative video 
log was on the first day of class when it was mentioned” (Thursday).” In this case, a leader who had proposed the 
idea provided a brief overview of the concept and answered questions raised by the group. The remainder of the 
meeting was spent negotiating logistics and expectations. Decisions were made about the expectations around the 
frequency and length of videos, the vetting process for quality and relevance of questions, and the choice to make 
every video public through YouTube. Members negotiated the day assignments, and a couple questions were 
formulated for starting the process. This meeting provided a “framework” for a shared understanding among the 
group. 

The members reflected that length, frequency, and number of participants in the CVL should match the 
setting and purpose for which it is being used. Members felt it was important for everyone in the group have an 
opportunity to pose at least one question to the group, and it would be “ideal to have each person have multiple 
times to introduce their questions” (Wednesday). Posting the videos online publically provided the group with an 
“authentic audience” beyond the artificial setting of the classroom; the members of the CVL perceived their public 
audience as a “big factor” in the quality of their experience. Curiosity was raised about the efficacy of number of 
participants in a CVL, with concern that too few members would create a challenge for engagement and flow of the 
project, where as too many would be overwhelming and less intimate. For this case, five members created a 
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consistent flow of videos during the week, for which the rate of production and observation was not overwhelming 
to any member of the group. 

Ownership 

There were many dimensions of ownership that were highlighted by members during their interviews. The 
most common of these were ownership of the topic, technological ownership, ownership of the CVL structure, and 
ownership of personal learning. The topic for the course served as a guidepost for the questions chosen by group 
members and was general enough to allow individual group members to focus on areas that interested them. One of 
the members commented that it “was really interesting how each of the members asked questions that were going to 
be helpful to them. I think part of the beauty of this is that we were genuinely looking for answers to questions that 
we had” (Thursday). Developing a question required reflection of the question’s relevance and scope. 

Additionally, it requires reflection as to how compelling of an answer others can provide to the question. One 
member stated, “I think it made me more selective about the type of question I asked. Knowing that, oh wait, I better 
have a good answer for this. So having to answer the question first influenced the type of questions I would ask” 
(Wednesday). This reflection of one’s own response as part of the question generation process is a demonstration of 
how self is the first audience of the vlogging process. Once set on a question she wanted to ask, one member shared 
that she would have “broaden[ed] the question to make sure everyone would have some way of answering” (Friday). 

This ownership over the weekly topics was highly valued by the group as a whole. Concerns were raised 
that if the instructors provided questions that it would have “squelched the creativity of it” and would have made “us 
feel like we just need to get this done.” One member felt instructor designed questions would have led to less 
authentic answers as people would have “preconceived expectations of how he wants us to answer” (Tuesday). The 
professor reinforced the efficacy of group generated questions stating, “Defining the questions, taking responsibility 
for that, was part of the project’s success.” Most of the group felt it would only be appropriate for an instructor to 
provide questions in settings where the maturity of the group warranted it. However, the members suggested that 
providing guidelines, example questions or topic areas could be helpful while allowing the group to retain control 
over the ownership of the topic overall. 

Technological ownership over the project was also discussed by all of the members. In particular, 
members expressed the value in being able to utilize their own hardware and software of choice. This provided 
group members the opportunity and access to these things, but also allowed them to utilize resources they would 
continue to have and could use after the CVL was finished. 

Each person found something that worked for that individual and I think that, had everyone been required 
to use the same software, it could have been a more frustrating experience for some people. So I don’t 
think that is something that should be tweaked. (Wednesday) 

There was a sense of the group’s ownership of the structure of the CVL. This sense of ownership was 
routed in the negotiation process of the various logistics and expectations of the CVL project described earlier in this 
paper. The terms ownership, freedom, and independence were frequently used to describe the control the group had 
over the topics and format of the group. One member stated, “It was kind of our thing and we ran with it” (Tuesday). 
This freedom was reinforced by the professor and teaching assistant. In reflecting on the project the professor 
commented, “You know, I wouldn’t have come up with it, I wouldn’t have, I did not have v-logs as one of the things 
on the initial list of possibilities that people could do. I liked that, you know, I like being surprised, and I like seeing 
people - given everything I’ve just said - kind of working outside the normal expectations.” 

Even though the CVL was a group project, members developed their own learning processes around their 
participation in the group. One member commented, “The interesting thing about the collaborative vlog project is 
that you are working with a group, but there is still independence in it” (Friday). Each member came to own their 
own processes for engaging in the CVL. Members reported having tracked their ideas through the week, written 
bulleted lists or full scripts outlining their video response, and actively crafted ways to present the materials through 
a visual medium. One member shared he “read them [his responses] to the computer screen,” (Monday). 

Finally, every member of the group reported participating in an ongoing reflection process that lasted the 
duration of the project. Commonly members reported reflecting on the topic during moments of downtime or after 
engaging some form of media. For example, Thursday shared, “as a newspaper reader, I would see things in the 
newspaper or hear something on the radio that would fit into the topic and 1 think here is some aspect related to the 
questions.” Sharing personal stories led to reflection, a common theme observed in the videos created for the CVL. 
This public sharing allowed for both the creator and viewer to reflect on the significance of the story resulting in a 
deeper appreciation of each other and the topics being discussed; this further demonstrated how self is the first 
audience in the vlogging process. Additionally, there were times when reflection led to realization, as stated by 
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Friday, “That was the beauty of having time to reflect because you started to realize wow, this is really an important 
question, and I should be thinking about this.” 

Academic Outcomes 

Gardner (2006) suggested open-ended projects provide students an opportunity to use knowledge from previously 
mastered concepts and skills to achieve a new goal and model the kind of work done outside of schooling in the 
community. 

It’s one of the unique things about this class . . . It’s a class that’s about technology, it’s a class with 
technology, it’s a class that’s taught through technology, and it’s a class that really requires students to 
develop new kinds of skills themselves, outside their familiar comfort. The things that I want are the people 
taking responsibility. Did they push themselves? Did they make something that was interesting and 
creative? Did they develop a new set of skills in the process of doing it? Did they work together as a group? 
Did they reflect thoughtfully on the process and learn from the process as well as what they learned about 
the substance or the content? Ah, 1 think your group did all those things. (Course Professor) 

Through using a collaborative process on a public venue to discuss pertinent issues about the class content, 
three themes emerged as academic outcomes in the data: building a learning community, realizing potential, and 
developing technological literacy. The first academic outcome was building a learning community. The members of 
the CVL were able to create a community of content creators who supported each other through the learning 
process. The participants described feelings of acceptance, appreciation, and comfort which arose from 
collaboratively taking a journey through a new venture. The students would help each other out by suggesting 
software, troubleshooting problems, and bouncing ideas off each other. Personal information was shared, which 
fostered a sense of community amongst the participants. The CVL allowed for members to identify an audience of 
four members, albeit a captive audience, that could potentially reach anyone who might come across the video on 
YouTube. One member described the public postings in the following manner: 

There’s still something about the possibility that anyone could view it as well that made the project 
interesting. I’m going to leave the door open. If anyone else wants to view, if they feel they can learn 
something from it, or if they’re interested by it, they’re welcome to. (Wednesday) 

The learning community for this project was made up of an audience that can be represented in a series of concentric 
circles that included self, the CVL, the class, and finally the potentially limitless audience on YouTube [see figure 
1]. Participants found they would reflect on their own lives, learning, and situations 

while viewing their footage throughout the editing and publishing process. Participants also found video was an 
effective way to communicate with each other that went “beyond simple words.” Learning was embodied through 
video clips, picture slide shows, and humorous cut scenes which allowed the video makers to express their own 
ideas that went beyond text based interactions typical of discussion boards and chat rooms. 

The second theme regarding academic outcomes was realizing potential. The CVL setting allowed students 
to completely formulate their answers and gave the students a voice and venue to share their thoughts. 

Many times I am sitting in class and there is a great conversation going on and in my mind I will have these 
thoughts, but . . . it just doesn’t fit so the conversation goes on to something else and I never made my 
point, and then I will think, a whole class went by and I really didn’t do a whole lot. I was thinking a whole 
lot of things but it didn’t really come out. Whereas with the video log, I had days and days to think about 
what you were going to say, and then you made your little video and you put it up there. I felt like it gave 
me the time to collect my thoughts and present them how I wanted on my terms (Thursday). 

The CVL was a catalyst that boosted confidence and inspired future interactions with video creation and research. 

All participants reported ideas in which they were planning on using video creation in their future endeavors that 
included informal social vlogging and formal situations which included teaching videos, professional reflections, 
and collaborative communications. The participants reported following through on some of their potential ideas like 
a musical CVL, video presentation projects, and online tutorials. These were made possible by the third theme: 
developing technical literacy, the most prominent theme when participants were asked about what they learned. The 
participants found creating videos much easier 
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Figure 1. The Audiences of the CVL 



than they anticipated at the beginning of the process. The vloggers experimented with video editing software that 
included text overlays, multi-media inserts, and video transitions. The participants found vlogging amongst their 
peers, who were all new to the idea of talking through a camera, a unique and novel experience. Friday commented, 
“I got a chance to play with it [vlogging] in a kind of non-threatening setting.” 

Collaborative Learning Environment 

A collaborative learning environment requires a safe space for individuals among the group to learn. “It 
was a conversation among the five people in the group who were all motivated to come to a greater understanding of 
the subject matter and share experiences with each other and gain new perspective” (Thursday). Members of the 
CVL described a sense of acceptance from the group they found to be both motivating and comforting. Thursday 
expressed her surprise about “how open and little judging there was.” She felt the same conversations face-to-face 
would have left her feeling like she had to “meet up to somebody else’s expectations.” Active participation among 
the group members also contributed to a sense of belonging and acceptance. Friday noted that not only did everyone 
participate, but they “seemed to legitimately invest in what they were doing, and that made me want to continue to 
invest at an equal or better rate.” Monday postulated that if there had been animosity within the group, it would 
have had a negative effect on the group dynamic. Thursday summed up the value of acceptance stating, “I can’t 
expect myself to give it more than I can. But I can expect my team members to be accepting of what I put out there 
and giving me feedback that will help me in my learning.” 

Building acceptance required participants to be accepting of diverse perspectives. Members shared how 
they learned from other members of the group and how “amazed” they were with the diversity in the perspectives 
brought to the group. The variations in perspectives were well received among members of this group. There 
appeared to be an authentic desire to understand the experiences of other group members and their perspectives. 
Thursday stated, “Hearing others input and take on the questions, spurred me to reconsider or even develop thoughts 
on most of these topics.” Friday shared how the experiences helped her to “rethink and recognize my own bias, and 
to better understand the depth and scope of the topic.” The questions posed by members spurred “conversations;” 
the topics were either so close to participants that they were typically taken for granted or fell outside of the topics 
traditionally discussed with their peers. Members frequently reinforced the arguments made by other members and, 
where possible, members scaffolded their perspectives with the perspectives of others building increasingly stronger 
and deeper arguments. 

The CVL design is unique in its ability to produce not only academic, but social outcomes as well. Three 
themes emerged which can be seen as ongoing processes which built off of each other [see figure 2], The 
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participants presented personal information to their audience on the vlog. Because the participants got to know each 
other, they started to embrace the differences they had. This led to expanding relationships beyond the classroom. 
The presenting of personal information appeared in a number of ways like developing one’s online identity, talking 
about one’s personal life, and deciding where to shoot one’s video. “One week (Friday) showed us her home 
remodel project, and 1 thought wow, that is cool. Where else would we have gotten that?” (Thursday). One vlogger 
commented on how well he got to know the others through the personal information they shared: 

With the collaborative vlog I know everybody’s name, what general vicinity they live in. I know if they 
have kids, if they have spouses, I know the kind of cars they drive, I know so much more about them then 
the other groups that 1 have worked with. Because of the vlog. Because there were just things we talked 
about that you wouldn’t write in a paper. That you wouldn’t talk about when making a poster for a 
presentation. (Monday) 


Figure 2. The construction of social outcomes 



Another distinctive aspect of the CVL is participants had to wait their turn to talk. They would share information 
about their own experiences and lives. But as a viewer, they had to listen intently to what their partners were saying. 
Wednesday said, “The nature of the format made it so that each person had a time where they were the one speaking 
and then everyone else actually listened.” Friday reflected, “You can't be playing in another screen. You can't be 
across the room just listening. You have to really be watching to see what people say, what they're passionate about, 
what's exciting, what's not.” This attention led to the vloggers getting to know each other in a way that may be seen 
as atypical of traditional classroom settings and asynchronous text-based discussion. 

By getting to know each other, the group started to realize how different they were and embraced their 
differences. This is one of the aspects that the originator of the CVL idea wanted to capture. Fie used the closing 
scene from the motion picture The Breakfast Club as an inspiration where the narrator discusses how the teacher 
sees each person as a stereotype (Flughes & Tanen, 1985); however, the students in the movie realize that each of 
them has elements of the others apparent in their lives as well. In an interview, he noted how he felt like an outsider 
in the class because he was the only one from his college. 

Before the project I didn’t know anyone. I am not in education as a major. It is somewhat of a minor for 
music education, so, all the people- It was my first class on campus since I have been here, for my 
doctorate, so I didn’t know anyone. (Monday) 

He was not alone. None of the members of the CVL claimed to know each other before the class started. 

Well before [starting the vlog project] we really didn't know much about each other. We all kind of came 
from different backgrounds with different goals in mind as to where they want to go. So that, that being 
said, we didn't really have a lot in common. (Tuesday) 

However, as the videos were created and viewed, the participants started to learn more about each other. Biases 
which would be considered normal in most classes seemed to give way to acceptance and even endearment. 

Two or three of the members of the group could have been my kids. That’s dynamic; the generational gap 
for me, I think brought a lot to me to see how they were thinking about [the topics]. And I think some of 
the thoughts might have helped them to get a different perspective on the topic. (Thursday) 

Their differences did not stop them from getting to know each other, nor did it hinder the learning process. 
In fact, the CVL provided a place for them to openly discuss their perspectives and celebrate their differences. The 
CVL format encouraged the formation of deep, meaningful relationships that were apparent in the classroom and 
extended beyond. The teaching assistant noted, 

I felt like that group was one of the more cohesive groups in terms of being on the same page as everyone 
else. ... I felt like when you guys had discussions in class, when you started contributing to different things 
you were sort of building off each other as opposed to saying well I disagree with that. (Course Teaching 
Assistant) 
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The group sought to be near each other in the physical classroom by sitting in the same section. Often times they 
would support each other in class or build off of each other’s ideas. However, this academic relationship was just the 
beginning. They sought each other out for conversations before and after class. “I think informally as we were 
getting together in the classes, we had little side bars, little opportunities to meet, little opportunities to share how 
things were going” (Friday). They would stay in the classroom after everyone else left to talk about personal things. 
“Usually we would hang around after the class ended and there were several times I remember us being the last 
group out” (Wednesday). The participants even shared how they would meet outside of class to catch up on a social 
level and enjoy each other’s presence. Finally, the group developed a sense of accountability with each other. 

Oh, definitely [relationships with co-vloggers impacted how I invested in the project]. The fact that we all 
were good friends, it made it not only that it was for a grade, but it was the fact that we all, well, I mean I 
guess even if you're not friends, even if you're just people in a class, I mean there's still a grade at the end. 
But this gives it a little more oomph or responsibility because of the fact that not only is it for a grade but if 
you don't perform, then you're going to let your friends down, not only just your colleagues. (Tuesday) 

The shift in relationship from classmates to colleagues to friends is apparent in all the interviews and video 
observations. The members of the group were allowed to experience a dynamic social change that is unique to the 
CVL model. 

Towards the end of the CVL project the group met face-to-face to create final video that featured the group 
together. The filming and presentation of the final video presented an opportunity for the group to participate and 
share through intentional group reflection. To facilitate the video the group had generated a group of questions 
which allowed them to a) reflect on the series events, their process, and their growth, b) what they had learned by 
being members of the group, and c) how they might use their new knowledge and skills in the future. Having been a 
supportive and accepting cohort throughout the CVL process, the participants created safe environment which 
allowed members to share openly about challenges and successes they experienced. Despite the familiarity that 
developed among the member, two members found their peers’ answers were at times “unexpected” and described 
the experience as “useful” and “fun.” It was also a reaffirming experience, as group members shared their 
perceptions of the project. For example, Monday shared, “My experience with the collaborative vlog group was very 
prideful I think. I was very proud of my group and to be a part of it.” This meeting and the final video provided a 
sense of closure for the project which allowed group members’ interpersonal relationships to move more 
successfully from primarily academic to social and professional. 

Drawbacks to the CVL 

The participants in this study reported a number of positive experiences and benefits. Though when asked 
about the potential drawbacks of the CVL, they listed the following: slackers, sensitive topics, non-age appropriate 
materials, lack of maturity in group members, animosity between members, access to resources, and group size. One 
drawback that was noted was the amount of time needed to produce a satisfactory video. The course professor 
acknowledged this challenge, “Record them, and edit them, and clean them up, and then post them. . . You know, to 
do it well takes time.” Similarly, being an audience member can also be time consuming. 

The time commitment [is a drawback of the vlog project], not so much for me in making the video because 
I was okay with how much time I put into making the videos. What was hard was finding the time to watch 
the videos, from the other people. Ironically, it is not more than 20 minutes to do but actually sitting down 
and doing it was a little tough. (Monday) 

The asynchronous nature of the CVL required participants to wait their turn to speak. While this allowed for the 
development of ideas and finding of voice, it also lead to frustration as an audience member. “The asynchronous 
nature could be a draw back. If someone does say something on video, there’s often times where I would love to ask 
someone, ‘Whoa, wait a minute, what do you mean by that?” (Wednesday). Finally, some members of the vlog 
experienced technical problems which were apparent through the analysis of the observation data. These problems 
included the following: a) video and audio was not synchronized; b) loss of words due to synchronization issues or 
transition effects; c) multiple upload attempts due to processing errors; and d) software limitations which were made 
apparent in a video that stopped abruptly in midsentence. While four out of the five members suspected there would 
be a steep learning curve, their actual experience is summed up representative comment: 

So, I guess initially, there’s a little bit of learning curve with using a video editing software, [long pause] 
And I’m hesitating because I kind of like learning new software and actually using it. So maybe I don’t 
want to see that as a draw back. But I would see others thinking that would be a drawback. Because that 
would require a little bit of time for people to sort of run with it and be comfortable with being a little 
uncomfortable, but after you start working and learning, that’s not an issue. (Wednesday). 
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Conclusion 


CVLs are a collaborative learning model designed to engage groups of students around a topic that 
promotes ownership, reflection, and academic and social outcomes. Members develop a sense of ownership that 
directs their learning. Reflecting on learning, self, and relationships leads students to realize their potential. A 
learning community is built which can cultivate social bonds that extend beyond the classroom. Concurrent research 
on the CVL is being conducted including a) looking at graduate students’ openness toward and perceptions of the 
CVL, b) how the CVL can develop voice and community in an undergraduate setting, and c) closer examination of 
formation of group and individual identity experienced by the participants in this study. By using online video 
technologies, the CVL has the potential to connect people while leading to dynamic, memorable learning. 
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Appendix A: Interview Questions 


CVL Group Member Interview Questions 

Opening Questions 

Please describe the collaborative video log project as if you were explaining it to someone for the first time. 

Can you describe your experiences working on group projects in other graduate level classes? 

Was your experience with this group project different? In what ways? 

Do you think there is a benefit in participating in group work in general? 

Group Learning & Dynamics 

What were you hoping to achieve or learn by being part of this project? Were you successful? 

What, if anything, did you learn by participating in the collaborative video log? 

Describe your social relationship with the other video log members before, during, and after the project. 

Is this typical of other group work you have experienced in life? 

Do you feel that the collaborative video log format helped to foster a social relationship with the other people in 
your group? 

Did these relationships impact how much investment in the project? 

Public Distribution 

Describe your personal interaction with YouTube as best as you can remember before you started this project? 
Did you feel about posting publicly to YouTube? 

Did knowing you were going to post publically impact how you created your videos? 

Did your perception of YouTube change over the course of working on the collaborative video log? If yes, how? 
How did video logging on an informal venue (YouTube) for a formal graded task affect the way you participated 
in this project? 

Project Structure 

How is using videos for a collaborative project different than other collaborative tools such as discussion groups, 
or online shared document such as google docs, pbworks, and wikis? 

How is using videos for collaborative project different than working collaboratively face-to-face on on a project? 
What do you feel are the benefits of a video log group project? 

What do you feel are the drawback of a video log group project? 

Topics and Answering 

Did you feel the group’s questions were relevant to the class topics? 

What was your process for coming up with questions for the video log? 

How did you feel about answering your own question first? 

How did you feel when others were answering your questions? 

How did you respond to learning other people’s questions? 

What was your process for developing your answers to the questions? 

How did having co-video loggers influence how you answered your questions? 

How do you feel it would impact the group dynamics if the questions were provided by the instructor? 

Closing Questions 

Did your perception of video logs and their purpose change over the course of working on the collaborative video 
log? If yes, how? 

Is there any part of this project that if it were changed would impact the project in a significant way? 

Do you feel a collaborative video log could be used in different educational settings? Where? For whom? 

Are there educational setting where a collaborative video log would be inappropriate? 

How do you plan on using what you learned in the future? 
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Instructor/Teaching Assistant Interview Questions 

Group Learning & Dynamics 

What do you believe is the value of assigning/participating in a group project? 

What were you hoping your students would learn by participating in group work? 

Can you describe the typical group dynamics for this course? In other courses? 

Would you describe the group dynamics you observed of the collaborative video log group? 

Do you feel that the video log format helped to foster a social relationship between members of the group? If yes, 
how? 

Did you perceive the social interactions of the collaborative video log group members as cliquey? Was this typical 
of other groups in this or previous classes? 

Do you feel the members of the collaborative video log achieved the learning goals you had for the project? 
Project Structure 

Please describe the collaborative video log project as if you were explaining it to someone for the first time. 

What do you feel are the benefits of a video log group project? 

What do you feel are the drawback of a video log group project? 

Did the format used by the group effect the way you graded? 

How is using videos for a collaborative project different than other collaborative tools such as discussion groups, 
online shared document (google docs, pbworks...), or wikis? 

How is using videos for a collaborative project different than working face-to-face? 

How does grading a collaborative video log differ from grading other collaborative projects? 

Topics and Answering 

Did you feel the group’s questions were relevant to the class topics? 

How do you feel it would impact the group dynamics if the questions were provided by the instructor? 

Public Distribution 

Why did you decide to require the projects in this course to be published on a public venue? 

What are the potential benefits and drawbacks of student’s posting their videos online? 

How did the group utilizing YouTube affect your perception of the final product? 

Closing Questions 

Is there any part of this project that if it were changed would impact the project in a significant way? 

Do you feel a collaborative video log could be used in different educational settings? Where? For whom? 

Are there educational setting where a collaborative video log would be inappropriate? 
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Appendix B: Data Collection and Processing Assignments 


Table. Data Collection and Processing Assignments 

Researcher 

Cayari 

Fox Wallon* 

Interviewer 

Monday 


X 

Tuesday 


X 

Wednesday 


X 

Thursday 


X 

Friday 

X 


Teaching Assistant 

X 


Professor 

X 

X X 

Transcription 

Monday 


X 

Tuesday 


X 

Wednesday 


X 

Thursday 


X 

Friday 

X 


Teaching Assistant 

X 


Professor** 



Coding 

Monday 


X 

Tuesday 


X 

Wednesday 


X 

Thursday 


X 

Friday 

X 


Teaching Assistant 


X 

Professor 

X 


* Robert Wallon is a researcher and member of the original 

CVL who actively participated in the collection and processing 

of the data for this article. 



** The professor’s interview was transcribed by Thursday who 

is another member of the Collaborative Video Log. 
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Creating Content on the Go: Innovative Mobile Apps for Teaching 

Huei-Lien Chen, Ph.D. 

Colorado State University 
801 Oval Drive #216 
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Woodbridge, VA 22191 

Descriptors: Mobile Teaching, Screencasting 


Abstract 

As mobile technologies continue to advance, faculty demand more support for creating content and 
immediate sharing through mobile tablet devices while on the go. Screencasting apps that allow lectures or 
interactive tutorials to be voice-recorded are powerful mediums for communicating ideas and delivering content. 
Educreations and ShowMe, two innovative apps, were introduced to faculty at two institutions. These introductions 
were followed by a study that helped faculty adopt these emerging apps in shaping teaching. 

Introduction 

The widespread adoption of tablet devices in addition to traditional desktops and laptops in schools has 
shifted teaching and learning into a new paradigm. Traxler (2009) stated, "Mobile learning technologies clearly 
support the transmission and delivery of rich multimedia content" (p. 17). To embrace 21st century technology for 
education, many schools have adapted the iPad as a 1:1 technology tool for teaching and learning as Apple 's iPad 
has maintained steady sales in the tablet market. Especially in higher education, more and more faculty are 
wondering whether there is a way to integrate mobile devices into their teaching while they are on the road, on 
sabbatical, or remotely accessing courses. 

Text-based lecturing is not optimal for visual and spatial learners. Screencasting, or classroom capturing, as 
it was defined more early on, has been integrated into online and blended teaching because of its powerful ability to 
communicate ideas and knowledge. As Harrison, Pidcock, and Ward (2009) suggested, narrated lectures can be a 
good tool to deliver course content to students. Donavant (2010) echoed, “The use of narrated digital presentations is 
one means of meeting learners ‘where they live’ and providing them with rich materials that will bring meanings to 
their learning experience” (p. 105). Screencasts allow students to learn in a combined audio-visual environment. 
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A Little History 


Since the software company TechSmith began developing screencasting tools, or applications, back in 
1987, synchronizing PowerPoint slides with audio and visual recordings has become more and more commonplace. 
Today, a PowerPoint presentation with embedded audio and or visual videos wouldn’t raise an eyebrow. 

Screencasting applications come either browser or client-based. Users who prefer browser-based 
applications can create an account on service websites like Screenr, Screentoaster, or Screenjelly to directly record, 
save, and share content via a web browser; meanwhile, client-based users first need to download and install software 
to their local device and then record and save the video as an editable version. The final product from that software, 
such as Adobe Captivate, Camtasia Studio, Jing, Snagit, or Echo360 Personal Capture, can then be published on the 
service website or distributed through video-sharing sites like YouTube or Vimeo. However, most of the above 
mentioned client-based software might not be supported by the IT department due to license fees, and although Jing 
has a free download, the five-minute recording limitation does not meet faculty’s lecturing needs. Furthermore, none 
of the applications are ready for mobile devices like the iPad. 

So, are there any screencasting tools ready to be used for mobile devices? The answer is yes. The authors 
discovered two apps for the iPad : Educreations and ShowMe. Both apps provide browser-based and client-based 
channels, which means that users could create narrated presentations not only through a preferred web browser but 
also through an iPad by downloading the free apps from the App Store. The instant sharing and hand-drawn 
annotation capability on imported images or photos could become very handy for mobile device users. (See Figures 
1 & 2. Note that both app interfaces might look different now, as both tools have been updated.) 

Generally speaking, Educreations and ShowMe are very similar in terms of use. Both apps make it easy to 
record, save, and publish. To summarize, Educreations can load multiple slides/scenes before starting and easily 
start a new blank page to add different content, but the use of undo/redo tools instead of the eraser tool is not 
intuitive for some users. In contrast, ShowMe has the eraser tool but does not load slides/scenes one at a time. The 
workaround to use pause and the eraser to create a new blank page is not convenient for users. 

A complete comparison table for these two apps is presented in the table below. 
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Educreations 

ShowMe 

URL: 

URL: 

www.educreations.com 

www.showme.com 

Cost: 

Cost: 

Free 

Free 

Channels: 

Channels: 

Browser-based/Client based 

Browser-based/Client based 

Tools: 

Tools: 

■ Done button (save lesson or start over) 

■ Close button (save or cancel) 

■ Colors 

■ Eraser tool 

■ Eraser (version 1 .4. 1 and newer) 

■ Image tool 

■ Hand tool 

■ Undo (while recording) 

■ Text tool 

■ Clear tool (clear drawings only or clear all) 

■ Image tool 

■ Record button 

■ Undo/Redo 

■ Pause/Record 

■ Delete all 

■ Colors 

■ Recording time 

■ Hide/show ribbon 

■ Record button 

■ Previous/Next page button 

■ Add page button 


Whiteboard: 

Whiteboard: 

Can write on multiple pages 

Can write on one page only. Need to pause the 
recording, clear all items on the screen, then import 
another image to continue the recording 

Import Images from: (See Figure 1.) 

Import Images from: (See Figure 2.) 

■ Camera 

■ Camera 

■ Photos 

■ Photos 

■ Dropbox 

■ Web 

■ Web 


Images Arrangement: 

Images Arrangement: 

One image per page; images can be rotated, duplicated, 

Multiple images can be stacked per page by using 

and locked 

move to front, send to back, and rotate features 

Multiple pages: 

Multiple pages: 

Yes 

No 

Narration: 

Narration: 

Yes 

Yes 

Number of videos and video length: 

Number of videos and video length: 

No limit 

No limit 

Publishing: 

Publishing: 

Personal and public viewing 

Personal and public viewing 

Viewing: 

Viewing 

Can be viewed on iPad and browser 

Can be viewed on iPad and browser 

Sharing: 

Sharing: 

Facebook and Twitter 

Facebook and Twitter 


Table 1. Comparisons of Educreations vs. ShowMe 
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Figure 1. Interface Screenshot: Educreations (Insert Image View) 



Choose Photo 


Take Photo 


Search images on web 


Figure 2. Interface Screenshot: ShowMe (Insert Image View) 
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Approaches 


To understand instructors’ experiences with Educreations and ShowMe, several Bring Your Own Devices 
(BYOD) hands-on workshops were provided at two institutions: Colorado State University (CSU) and Northern 
Virginia Community College (NOVA). 

BYOD Hands-on Workshop 

During the workshop sessions, observation notes and workshop feedback were collected: 

a. Observation Notes 

b. Workshop Feedback (Google Forms) 

i . comfort level before and after the workshop 
i i . expectations for the tools 
i i i . concerns for the tools 
iv. anticipated uses for teaching 
Follow-Up Survey 

A follow-up survey created with Google Forms was distributed a week or two after the workshop. Three categories 
of data were collected: demographic data, teaching experiences, and screencasting experiences with Educreations 
and ShowMe. 

Research Questions 

Question 1 : What were your expectations for Educreations and/or ShowMe ? Did the workshop meet your 
expectations? 

Question 2: What are your concerns about Educreations and/or ShowMel Will those concerns affect your use of the 
tools? 

Question 3 : Do you plan to use either Educreations or ShowMe for teaching? If yes, please share your ideas of using 
them. If not, why? 

Question 4: What was your takeaway from the workshop? 

Our Experience 


First Site (CSU) 

BYOD workshops 

■ Eighteen (18) faculty members signed up for the workshop, but only nine (9) attended. There was a 
mix of male and female faculty members, including two female doctoral students. 

■ Four (4) faculty members have been teaching online courses, but others have not. The two (2) PhD 
students only teach residential courses. 

■ Lecture capturing tools CSU provided included: Camtasia, Captivate, Echo360 classroom capture and 
personal capture. One faculty member used all three tools to present multimedia content. The others 
seldom use lecture capturing tools. 

Observations 

■ Several participants did not know how to connect to the university’s WiFi network using their own 
iPads during the workshop. This indicated that they use the iPad mainly for personal uses and might 
not use it at work. 

■ Some participants requested assistance from the presenter or other participants. 

■ One participant did not know how to save a lesson. 

■ There were gender differences: 

o Female participants had more problems logging into the college’s WiFi connection, 
o Male participants were more comfortable using the iPad apps and thus felt more comfortable 
sharing ideas. 


Second Site (NOVA) 

BYOD workshops 

■ Four training sessions were offered to faculty and staff. In total, ten (10) participants attended the 

training, including one (1) administrator, three (3) staff members, and six (6) faculty members. Faculty 
members’ teaching areas include Math; English, Writing, and Literature; Biology; ESL; and Massage 
Therapy. 
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■ Six (6) participants had been using the iPad for a while and were looking for new apps that could be 
used for their teaching while four (4) other participants had their brand new iPads at hand and were 
curious what apps are all about. 

■ NOVA provided two lecture capturing tools: Blackboard Collaborate and Panopto. Some of the 
participants have been using those two tools to record their lectures. 

■ Five (5) faculty members teach online or hybrid course and only one (1) faculty member teaches face- 
to-face courses. 

Observations 

■ Some participants needed help connecting to the school’s WiFi. 

■ Some instructors asked how to project an iPad screen in the classroom. 

■ Participants felt frustrated when they couldn’t upload multiple images at once. 

■ All participants enjoyed the training with various hands-on practices. 

■ Participants felt highly motivated when the facilitator switched the projector to their iPads and let them 
demonstrate their videos at the end of the training. 

Results 

Question 1: What were your expectations for Educreations and/or ShowMe? Did the workshop meet your 
expectations? 

"I did riot really have any expectations, and I learned how to create a video, save the link to RamCT and screen - 

capture my iPad display. ” 

All (19) participants felt very comfortable with the apps after the training, and some of them indicated their 
confidence level and ability in teaching others to use these apps. 

Sixteen (16) participants felt the apps met their expectations. The participants could upload images, 
recorded their voices over the images, and create a video to share. Three (3) instructors had been using Panopto, 
Camtasia, or Blackboard Collaborate, so they expected the apps could record the action instead of just still images 
on the iPad. 

Question 2: What are your concerns about Educreations and/or ShowMe? Will those concerns affect your use of 
the tools? 


■ One faculty member wanted to know if he could use the iPad as an input device while recording the 
lectures and embed quizzes using Camtasia. 

■ One faculty member wanted to know the image resolutions for both apps, and if both apps accept 
vector images instead of the bitmap. 

■ One participant expressed the biggest concern for the apps is whether the video is secure, especially 
when the recording content might reveal students’ confidential information. 

■ Another concern is whether students can easily access recordings. 

■ Students who don’t have an iPad might feel left out if they were asked to create their own videos using 
iPad apps. 

“Compared to what I can do on my laptop with Camtasia, [the iPad apps] are very limited; however, for a free app, 

they worked fine. ” 

“If [the app] could record the continuing motions like desktop application Jing instead of only the still pictures, it 

could be very’ useful and time saving. ” 

Question 3: Do you plan to use either Educreations or ShowMe for teaching? If yes, please share your ideas of 
using them. If not, why? 

Faculty members believed the apps could be used to conduct a quick explanation to answer students’ 
questions. Biology faculty members suggested giving each group of students a topic, and then they could use the 
built-in camera to capture the real life image and add the annotation to the images. By the end, each group could 
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share their projects with other groups. English and ESL faculty members proposed engaging students by asking 
them to record their own voice over the images for literature and reading/writing courses. Math instructors and 
economics faculty members were very happy with ShowMe because it easily recorded all the steps of problem 
solving. In addition, the staff members and the administrator showed the highest interest in these apps and planned 
to use the apps in their daily work. 

“I will make some short videos and test them with my students. ” 

“I anticipate using Educreations for professional workshops when I travel more than for class due to the limitations 

with uploading and formatting presentations. ” 

Question 4: What was your takeaway from the workshop? 

“ The two big things I learned were 1) How to make a screen capture on my iPad. lean make a screen capture, 
import it into Educreations, and then highlight parts of the capture. 2) How to put the link for my video into 

RamCT. ” 

“I need more practice using the software. ” 

“ I realized that the challenging part is not the application itself but the idea of integrating it into the lesson plan and 

learning activities. ” 

“This could be a very’ good tool for students ’ group work and clearly present their work online. ” 

Faculty exchanged ways of using these apps for teaching: 

■ Answer student assignment questions 

■ Provide illustration and annotations 

■ Provide instant recording while in the field 

■ Provide instant feedback for students on assignments/projects 

■ Encourage students to create their own projects, such as a concept map, a flow chart, etc. 

■ Provide quick instructions for a certain concept 

■ Motivate students by inviting them to create their own videos 

■ Share videos from mobile devices easily 

Limitations and Challenges: 

■ Not working on all file formats 

■ Can’t import all images at once 

■ No shape tools 

■ Not ready for Android devices 

■ Can’t embed quizzes 

■ Can’t record screen motions 

■ Accessibility issues (no closed captions capability) 

■ Technology divide (not everyone has an iPad) 

As a side note, although using iPads for teaching is popular, it is notable that most of the participants 
lacked the knowledge to do basic operations with the tool. For example, most participants did not know how to 
make a screenshot with an iPad or how to copy and paste a URL into a message. Inevitably, we encountered some 
difficulties recruiting more faculty members to sign up for the workshop. As the administrator participant concluded 
after the workshop, “I wish more faculty [members] could learn about these apps.” 

Discussion 

We have not yet compared the two institutions in regard to the impact of student learning with these iPad 
apps. The impact varies greatly, depending on the course, products, and skill set of the instructors and students. 
Donavant (2012, p. 118) noted, “Narrated digital media presentations provide a viable and convenient means of 
conveying educational material in many settings, but they are particular useful in an online environment, where 
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facilitators often struggle to present information in rich and meaningful ways.” The participants did state that they 
feel the apps can provide quick instructions and answers in a feasible and convenient way. 


The conclusions do not suggest that iPad screencasting apps are better for instructors, whether it is in an 
online environment, face to face, or in a blended setting. These results just demonstrate why more research is 
needed. There is much more to be learned through our study. Other important considerations include student 
engagement and whether or not the instructional materials created with these apps produce positive results. 

However, the apps won’t be available for students to use until faculty members learn how to use them and can then 
provide guidance and require apps to be used for homework or group projects. Successful education has always been 
about engaging students. There are a number of factors that influence the efficacy of online or blended learning. For 
these apps to be successful in an educational setting, it is fundamental and essential to have faculty be fully engaged 
and supported in the use of instructional technology. 
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Abstract 

This paper presents the views of four authors who attempted to answer the two seminal questions: 1) What 
is a quality education? And 2) What does it look like? Rather than serve as a research paper, however, this is written 
as an opinion piece intended to generate discussion and dialogue on this topic. Does systems thinking and human 
performance technology represent appropriate mechanisms for helping refine our current k-12 education system? 

Our authors sound off on this topic. 

Anthony Chow, Ph.D. - Educational psychologist, performance technologist, educational reformer, former school 
board candidate, and father of three (one elementary, one middle, and one high school student) 

What is a Quality Education? What does it Look Like? 

First and foremost, like any quality instruction, this is subjective to the individual learner. What are their 
values? What are their goals? How does education compliment the world view of a student’s parents, culture, local 
community, and society? In essence, the systems view of thinking and developing instruction would start with, 

“what are the intended outcomes and accomplishments we want the learner to be able to do and achieve at the end of 
instruction?” and then work backwards (backward design) from there. 

Right now, most schools focus the majority of instructional time and assessment on learning academic 
content and skills. This, however, is not consistent with Joe Harless’ (1998) view of how one develops a high quality 
performance technology system or educational experience. The reason why is if you ask parents there are much 
higher level accomplishments we want from our children. Same could be said for employers and society in general. 

What do I mean? Well, we certainly want our kids to be smart, get good grades, score well on tests, and go 
to a good college so that one day they can become economically independent and contribute back to society. This is 
a traditional middle class worldview that is, however, not necessarily shared by everyone; more on that in a moment. 
My dissertation topic studied the school that Joe Harless (1998) helped develop based on his performance 
technology principles and ABCD model (Accomplishment Based Curriculum Design). When he posed this question 
to his local community what he got back was a strong work ethic that embraced respect, hard work, honesty, 
integrity, team work, communication, and was centered almost entirely on a being an accomplished citizen in a free 
thinking society. In other words the ideal school (they were going to build) needed to focus on helping students 
build these work ethic schools and the rest of the facets traditionally associated with success would follow - good 
grades, a good job, a quality of life that is paved out through hard work, working well with others, and being a good 
citizen. The emphasis was not on doing well in school or going to college. It was simultaneously at a much more 
basic and higher level than that. School was just a means to a greater end. 

Right now in the headlines (as of October 23 rd , 2013) we are reeling from the recent deaths of two teachers 
from two separate incidents of school violence across the country. Bullying is associated with one of them it would 
appear and the other we did not know yet what happened. Bullying and suicide has also been in the news, which 
suggest many of our students have current needs that extend way beyond traditional academically driven and 
focused content. These students need more than drill and practice and discussion around math and English and 
arguably more emphasis placed on getting along with one another, values, and learning how to be a healthy human 
being in a very complex world. Being an adolescent further complicates and increases the degree of difficult (and 
necessity to address these issues) exponentially. 

The school that was created by Harless and his colleagues was the Central Educational Center (CEC) that 
focused on career and technical education (CTE) because career-oriented instruction provided the performance and 
accomplishment-based qualities Harless and the local stakeholders wanted out of their children’s education - clear, 
project-based outcomes that occur in a team environment, which requires lots of communication, teamwork, and 
rather complex academic skills that were applied in actual performance settings. In other words, work-ethic grades 
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(assigned by technical colleges for decades) were as important, if not more, than the traditional academic grades and 
were also hands-on so that students could see how their learning was being applied immediately. 

Back to the question of individual learning, however, this CTE school is a charter school and therefore a 
school of choice, which is an essential ingredient to ensuring learner motivation is present. Parents and students 
choose to attend this school and therefore overall performance is much higher because the students choose to be 
there and choose to take the classes they are taking. 

What does this mean for the traditional curriculum? The movement to common-core standards is a large 
step in the right direction as it is focused more on the applied rather than just factual-based and procedural 
knowledge. What I think needs to occur, however, is that the social development and health (emotional and 
physical) of students needs to move to the front and center of the traditional curriculum. I call them “here-and-now” 
needs where students are being taught basic psychology, communication and teamwork is made the highest priority, 
and work ethic is one of the fundamental core subject areas that integrate with the traditional big four of math, 
language arts, science, and social studies. 

Let me give you a personal example. My son has been bullied most of his school experience and it has 
indelibly shaped his personality (in a negative way) and has caused me to revisit my own personal values and ability 
as a father and care-giver. Have I raised him correctly? Have I done something wrong? How come the schools he 
goes to do not pay more attention to this? After careful examination, 1 hold myself accountable for not teaching him 
the verbal and social skills necessary to not be one of the kids that bullies target. As a hard working professor, 1 live 
on a daily basis the type of work-alcoholic tendencies necessary to be successful in my chosen career. He has tried 
to emulate this. But at the same time, ironically, the very values that I personify and hold dearest personally, 
professionally, and culturally are the prominent reasons why he is being bullied. You see my son is doing exactly 
what we have asked him to do and raised him to be - smart, quiet, hardworking, humble, and nice to people. In other 
words, we have taught him to be what society views as a traditional “geek.” When you really think about it if people 
like my son are eschewed by society as a “target” or “weak” when I view this as being a good, accomplished citizen 
then we have a serious problem. 

In my opinion, this is exactly why our schools must place much more time and emphasis on teaching 
(really demanding) our students practice every day the basics of good communication, teamwork, work ethic, and, 
essentially, good citizenry. Those who bully need to be idendified and helped - bullies reflect deep insecurities that 
compel them to cause pain to others in attempt to make themselves feel better. This is empowering for them and the 
wake they leave is harmful in every way to everyone they are around. When my son experienced bullying at one of 
our school district’s top performing schools, I communicated with his teachers and principals, that bullying should 
not be tolerated and must be addressed by dealing directly with the bullies and also ensuring that all students do not 
tolerate it. Back to my son who I hope can forgive me for this later on - he had two girls mercilessly harassing him 
both in class and in the hallways. My son is a pretty big guy and does very well in his classes but the verbal abuse 
and even physical bumping and tripping was starting to get to him. “Dad I hate school.” “Dad I feel horrible about 
myself.” “Dad, why bother working hard because I’m stupid.” Eventually, after the beginning of his second 
semester as a freshman in high school he finally confided in me about what was going on. I immediately turned to 
his teachers and principals and we collectively rectified the situation - neither of the bullies is still at his school as it 
was clear to everyone that they were not a good fit for this school culture. His confidence and school performance 
did a 360 turn and he is now the happy, healthy kid we want most of all as parents. 

With this as a context, let me attempt to answer this question from multiple perspectives - as educator, as 
parent, and as a citizen. 

As an Educator 

A quality education means that each individual student is learning and improving based on his/her starting 
point. In other words, the emphasis on norm-referencing needs to take a back-seat to individual improvement and a 
trajectory towards individualized instruction and competency. Emphasizing comparisons amongst peers creates 
winners and losers and those that score below the norm usually are demoralized and lose further interest and 
motivation for school. 

Systems thinking and the ADDIE model suggest that schools must establish long-term outcomes that serve 
as clear, measurable goals for education. This type of thinking works for small scale training sessions, whole 
educational programs, organizations, and performance in general. The current emphasis on content standards that 
drive public education does not necessarily reflect the overall expectations of parents, employers, and society in 
general. The good news is that systems thinking allows for local communities and their primary stakeholders to 
decide this. It must be a process that every local community must go through. 
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Content specific standards are also not well aligned with traditional learning theory (e.g. Gagne’s Theory of 
Learning) that posits gaining a learner’s attention and establishing and maintaining motivation towards the subject 
matter and content being introduced are the most critical first steps to learning. Project-based learning such as what 
Harless did with career and technical education reflects to a higher degree what accomplishment or performance- 
based education is all about - learner’s will be able to complete some kind of task (not take a test) that has been 
identified by society as critical for both the student’s and community’s future. Following this pathway would allow 
for educators to teach the fundamental core areas in a myriad of applied settings, which most often will be work 
environments or specific problems faced by society that need to be solved. Content and the skills they reflect are 
really a means to an end, which is performing some kind of meaningful skill. 

I often mention to colleagues that the “nerd” syndrome in our society is a testament to a major failure of 
our education system. The joke is that some of our best and brightest and hardest working students that have high 
GPAs and standardized test scores have little to no “common sense” or ability to do “meaningful” things that help 
make them competent individuals in the “real-world.” Pretty ironic that our schools cannot teach bright, 
hardworking, and motivated students a curricula and skills that are “practical” and “meaningful” in their day-to-day 
lives. Another issue of course is that knowledge and intellectual capital in today’s k-12 schools are often looked 
down upon in favor of sports and other “cooler” activities. 

As a Parent 

I have three primary goals for my children: 1) To be happy in their lives, 2) To be successful in achieving 
their goals, and 3) To be productive and competent members of society. Much of their life lessons and establishment 
of individual values occur at home but at the same time they spend many of their waking hours in school. As a 
developmental psychologist, it is self-evident that the impact a school climate has on youth, their emerging 
identities, and overall emotional health while going to school (from k-12 or 5-19 years of age) is a pretty major 
variable. 

To limit the conversation to only traditional academic content and not pay more attention to a school’s 
organizational culture and the values they represent and the impact this culture has on everyone involved - 
administrators, teachers, students, and parents - is leaving things primarily up to chance, which rarely works out 
well, is sustainable, or consistent. I want my children to be happy, successful in achieving their goals, and be 
competent and productive. In other words I want them to have a strong work ethic, be able to work well with others, 
and have the ability to take care of themselves, establish clear goals, and have the know-how and discipline to attain 
them. 

Bullying and a sole emphasis on checking-off a litany of courses in the main core academic areas are not 
highly aligned with these goals. In the work place, if someone pushed you, bullied you, hit you, harassed you 
systematically, they would be fired, arrested, etc. This kind of behavior, along with quitting school, losing interest in 
school, and performing poorly over a long-term period of time also is not acceptable. As an instructional 
technologist, any student who fails reflects a failure of the system. 

As a parent, I want to be on the same page with my children’s educators that we collectively are helping to 
raise hardworking, respectful, and well-educated kids that work well with others, are learning things at school that 
are both applicable in their here-and-now real world context and also on the path towards college. I care less about 
them being straight-A students and high test scorers and more for their happiness, positive attitude, and having the 
skills necessary to be highly competent, productive members of society. 

As a US Citizen and American 

The statistics are pretty compelling that high school drop-outs are much more likely to be incarcerated, be 
unemployed or underemployed, use drugs or alcohol at higher rates, etc. than those who graduate with a high school 
degree. In other words, the cost to society for every individual that drops out is extremely high both for the student 
and the community in which they live. Basic systems thinking would tell you that any organization or system that 
has a high failure rate is not working properly and has many opportunities for improvement. Studies on drop-outs 
show that it is a gradual and long-term disillusionment with the curriculum, formal schooling, etc. that is clear for 
everyone to see. While the solutions are not easily identifiable as the factors behind drop-outs vary, the evidence is 
pretty clear that public education is not working for a lot of our school-aged youth, their families, and local 
communities. 

As a manager of such a system the first question we must ask and have answered is why is the student 
disillusioned and what can be done to help change that? On the university level, a growing trend is to hire adjunct 
professors whose sole job is to provide direct support to all of a department’s students and pay special, hands-on, 
and personal attention to those who are struggling and showing signs of dropping out. With the growing use of data 
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and big data to analyze trends and provide analytics and informatics that helps organizations identify and respond to 
such “red-flag” indicators as student drop-outs and at-risk students, the time has come for us to begin addressing 
these issues and others such as emotional distress, mental health issues, etc. that are non-curricular but very 
important for the well-being of our nation’s students. 

As a citizen I want hardworking, respectful, and competent high school graduates who have the 21 st century 
skills necessary to be happy, successful, and competent citizens of our global economy. 21 st century skills must 
incorporate previously considered “soft skills” such as self-regulation, work-ethic, teamwork, communication, etc. 
Current problems such as high dropout rates, bullying, a curriculum that is too academic, and an under-emphasis on 
the wonderful qualities of just being “nice” and “smart” need to be addressed. 

Systems thinking would help identify and establish clear systemic pathways for making this happen. As 
many have said, the paradigm of public education must shift based on the needs of the community schools serve. 

Kenneth Prest, Ph.D. - Fishery > biologist, ecologist, environmental engineer, inventor, information management 
consultant, and father. 

What is a Quality Education? 

It depends.. . .on how we frame the concept of “education” in our minds; what we want from the education 
we’ve envisioned; and how we define the nature of the gap between what we want and what we’ve got. 

In my thinking, “education” is not an “end” but a “means.” It is the process of recursively producing the 
selves that we need to be to live in the environment — of people, things (natural and cultural), and information — 
within which we find ourselves at the time. 

Education is the forward-looking, life-spanning process of continually developing and redeveloping 
knowledges and skills — specific and contextual; concrete and abstract; individual and social — that enable us to 
adaptively make connections, facilitate transactions, and build and sustain relationships that matter. Whatever our 
social roles or circumstances, education is about becoming intentionally informed, enabled, and, ultimately, deeply 
engaged within the ever-evolving complex adaptive systems we call life. 

In a systems context, educational enterprises would create educational experiences that were intentionally 
and tightly coupled in the world. Both enterprise and experience would be defined by a focus on: (1) creating real 
long-term value for stakeholders — the customer, supplier, and community beneficiaries of the education; (2) an 
alignment of operational ends and means; (3) an integration of interests of members working to create the 
educational experience on the inside with the interests of stakeholders benefiting from that experience on the 
outside; and (4) feedback loops to enable both the enterprise and the experience to be readily adjusted as knowledge, 
skills, tools, applications, and people change. 

As framed and focused by vision, alignment, integration of interests, and feedback, a “quality education” 
would emerge as the enterprise defines itself: To what ends at what organizational performance “level” by what 
means under what circumstances? 



To what ends at what organizational performance “level”...? 

A. Society (Unity defined by city, county, state, or nation political boundary or by world 

physically constrained) 

B. Community (Unity emerging from integration of individuals, families, and organizations with 

a shared history) 

C. Organization (Unity’ differentiated as government, business, or civil society organization) 

D. Family (Unity of individuals living together) 

E. Individual 

By what means...? 

A. Public Education (Focus on laying the foundation of performance knowledges and skills — 
specific and contextual; concrete and abstract; individual and social — that are common to all 
organizations because they are core to the life processes of making connections, facilitating 
transactions, building relationships that matter among people, things, and information in 
minds, markets, and world.) 

B. Private Education (Focus on building on top of the public education foundation by 
customizing or supplementing the performance knowledges and skills for a specified result in 
context of an intended outcome — whether in government, business, or civil society’.) 
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III. Under what circumstances. . . ? 

A. Education Levels (Reflect incremental learning process “starting points ” and “value-adding 
segments ” through which we educate ourselves — individually and collectively; formally and 
informally — subject by subject, over a lifetime in a continuous flow.) 

1 . Elementary 

2. Intermediate 

3. Secondary 

4. Post-secondary 

5. Adult 

B. Learning Frames — (Frame all learning in terms of (1) the life process outcomes of 
connections, transactions, and relationships that matter among people, things, and 
information that are brought forth in language that, in turn, emerges from applying 
technology’ innovatively, managing information purposefully, communicating effectively, and 
living and working with others collaboratively — in short, by “information management ” — 
and (2) the fields of endeavor that make the life process outcomes meaningful.) 

1. Life process outcomes (Connections, transactions, and relationships that matter 
within minds, markets, and world.) 

2. Fields of endeavor (Subject matter areas across the arts and sciences through which 
life processes meaningfully operate.) 

C. Learning Perspectives (Emphasize both theory’ and practice recursively engaged through 
research to perpetuate self-developing, knowing-doing abilities across fields of endeavor, life 
processes, and lifetimes.) 

1. Experiential (Research) 

2. Conceptual (Theory) 

3. Applied (Practice) 

D. Performance Areas (Align learning with the way the individual works in a nested and 
networked world — concurrently focused inwardly on the task in context of the value created in 
relationships that matter to the team and focused outwardly on the team in context of the 
value created in relationships that matter to stakeholders.) 

1 . Personal knowledges, skills, and abilities (Enable the individual to create value in 
knowing, deciding, and doing things related to the task.) 

2. Interpersonal knowledges, skills, and abilities (Enable the individual to create value 
through knowing, deciding, and doing things related to the team.) 

In sum, “education” is effective action in an ever-changing world. Within a global society of increasingly 
diverse, interacting individuals and organizations, our ability to produce, adapt, and sustain ourselves is a function of 
our ability to act effectively in creating valued outcomes in relationships that matter to families, organizations, and 
communities within which we are engaged. 

In my mind, a “quality education” emerges as educational enterprises catalyze a life-long educational 
experience through which we continually learn how to learn to make connections, to facilitate transactions, and to 
build and sustain the relationships that matter regardless of the environment within which we find ourselves. 

So, “What is a Quality Education?” My simple answer is: The essential tool we — as families, 
organizations, and communities — intentionally deploy in order to thrive, not just survive, in a pluralistic, rapidly- 
changing, nested and networked world. 

Beth Raj an Sockman, Ph.D. - Professor, instructional technologist, mother, and educational reformer. 

1) What is a Quality Education? 

A quality education can only be completely described within the time period addressed as it is determined 
by the beliefs and values held within society. It is dependent on what we know about learning at any particular time, 
Luther characterized by the tools available to support the learning environment. 

Since society needs drive what is defined as quality education, education should provide the foundation so 
that individual persons can survive and thrive within the given economy. Being a product of the United States, 
thriving means that that education should provide the access to skills and processes so that the learners have tools to 
the betterment of “Life, Liberty and the pursuit of Happiness” (Declaration of Independence, 1776). Quality ensures 
that very learner has access to be engaged in stimulating environment for that time. 
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At this particular time period, most agree that students learn in different ways and at different times that we 
need to prepare learners to thrive within a democracy. A quality education will emotionally and academically 
support learners within the learning environment. To function within a democracy, there needs to be a respect for 
differing values and perspectives along with supportive learning in communication, creativity and problem solving 
skills within the domains of language, numeracy, science, arts, and civilization development. 

2) What does it Look Like? 

A quality education embraces what we know about learning at this time, and uses the tools available to 
create the learning environment to best support the process. This process begins with wise educators that value the 
uniquely wonderful learning differences within each individual. This acknowledgement of differences leads to the 
need for learner customization. 

The educator support and tool support engages learners within the domains of language, numeracy, science, 
arts, and civilization development, which are further divided into the basic skills, problem solving and creativity. 
This requires individual learners to learn basic skills at a pace and means which is best for them. We know that 
people learn in different ways and different times, but too often education’s structures as if all people learn at the 
same way and at the same time. If students learn these basic skills in different ways, then various technology 
including computers, manipulative, and pacing resources should be used to guide the process. 

Within the content domains there should be significant tasks that require learners to use their skill 
knowledge to creatively solve problems. These problem-solving tasks are often cross disciplinary and require 
collaboration with others. Throughout the process, individuals should engage in group deliberations a so that they 
learn to communicate about the domains, seeking to understand other perspectives and refining their own 
perspective. Krishnamurti said, “the very instruction of these subjects is to bring about a change in your (student’s) 
mind” (1974 Pg. 126). Student subject or domain learning should not just be for the regurgitation of information, but 
so that learners can see the world differently. Ultimately, learners should be making the world a better place for that 
upholds the equality of humanity. 

Frank Duffy, Ph.D. - Professor, scholar, community leader, father, and educational innovator. 

What is a Quality Education? 

For me. ...a quality education is one that prepares children to live a successful life. 

A quality education is one that identifies the unique needs, interests, and abilities of each child and then 
helps each child learn what needs to be learned by providing him or her with learning opportunities that are aligned 
with his or her needs, interests, and abilities. 

To Discuss This Topic Further 

To discuss this topic further please go to http://svstemicchange.wordpress.com/ and a discussion area has 
been created for this paper and presentation at the 2013 AECT Conference. 
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Abstract 

The purpose of this study is to explore a cell-phone Short Message Service (SMS) -based immediate 
feedback system and to reveal opinions of instructors. This study is designed as a qualitative one and data were 
collected via semi-structured interviews with 4 instructors who used the system in their classrooms. The significance 
of this study is to help to remove obstacles about the design of such cell-phone based immediate feedback systems 
and exploring pedagogical guidelines/principles. 


Introduction 

Any mobile phone which has the ability to send messages could allow teachers to quiz students, to assess 
students’ prior knowledge of a topic and to help determine patterns of thinking in the classroom and all these 
purposes aim to support increasing classrooms interaction and the active participation of students. The use of the 
mobile platform as a student immediate feedback system has the same logic as clickers. For that reason, the mobile 
platform as clickers includes the benefits of classroom clickers. (Kaleta & Joosten, 2007; Hoekstra, 2008; Bojinova 
& Oigara, 2011; Lennox Terrion &Aceti, 2012). 

The purpose of this study was to explore a cell-phone sms-based personal immediate feedback system and to 
examine the instructors’ opinions and requirements about the system. Additionally, based on their opinions and 
suggestions, this system will be iteratively developed, tested and evaluated. The significance of this study is to help 
to remove obstacles about cell phone sms-based personal immediate feedback systems in addition to supporting all 
the benefits provided by personal immediate feedback systems. 
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Methodology 


The system is managed with a web interface. Instructors log in to system with their user names and 
passwords. After logging in to system, they are allowed to write and ask their questions. The system gives 15 
minutes as default response duration for each question. Students only send their selection via text message. System 
automatically sends a confirmation message to the students, whenever their selection is received. 

Context 

The system was used in four courses. One of four courses was a graduate level course; three of them were 
undergraduate level courses. While, the number of students in graduate course was 14, for the other undergraduate 
lessons, the numbers of students were as following: 20, 27 and 65. 

Participants 

Participants of this study consisted of four faculty members from different departments (computer education and 
instructional technology, educational science, civil engineering and industrial engineering). Three of them were 
female, one of them was male. All faculty members used clicker system before. 

Major Findings 


Advantages 

Anonymity: One of the instructors mentioned that the system being anonymous encouraged students to engage 
activity. 

Availability: Three of the instructors stated availability as an advantage of cell-phone SMS-based immediate 
feedback systems. Especially one of them emphasized that cell-phone use makes it possible “everywhere” and 
“anytime” interaction without any extra equipment. Furthermore, using cell-phone removes possibility of swapping 
devices that cause identification problems regarding grading and absenteeism. 

Crowded Classrooms: Two of the instructors said that the system is appropriate to use in crowded classrooms 
rather than small size ones. The system was found to be fast and practical especially for crowded classrooms. For 
instance; one of the instructors collects responses of students with piece of papers and she claimed that this wastes 
papers unnecessarily. So, the system may help to overcome this problem both from the point of wasting time and 
source. 

Limitations 

Time Consuming: Three of the instructors stated that using this system was time consuming. Because, while two of 
them mentioned that preparation of multiple-choice questions took too much time; two of them indicated that 
response duration of creating graphics and representing results lasted long. 

Design Issues: This limitation is given under three subtopics which are presentation options, showing results before 
voting finishes and one screen for all. Two of the instructors stated that they preferred to see the results on a bar 
graph instead of a pie chart. In addition of this, they would like to have an option to choose type of graph. Three of 
the instructors indicated that they had to use a second screen to show question and choices. Additionally, they 
wanted to present the question, choices and the phone number that students should send responses on one screen. 

Novelty Effect 

This topic is found to be not only an advantage but also a limitation. So, novelty effect addressed as a separate topic. 
Three of the instructors stated that students were excited to use the system. The instructors observed that students 
engagement and interaction in the classroom increased. Moreover, the instructors mentioned that this attitude of 
students came from novelty, and one of them especially emphasized this situation as “novelty effect”. However, 
whether these engagement and interaction were only caused by novelty, or not is ambiguous. 

Suggestions 

Two of the instructors suggested that the system should be integrated with student affairs information system, 
because they want to use it for absenteeism and grading. One of the instructors proposed that the system may report 
overall questions and results for each one in addition to statistical results. Moreover, in order to not to waste time in 
class due to writing questions to the system, one of the instructors offered that the system should be allowed to input 
the questions previously and activate them just before asking. One of the instructors asked for keeping all data to 
investigate students’ conceptual change during the term. Additionally, one of the instructors suggested that the 
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system should let instructors choose anonymity or onymous to explore students who give incorrect answers 
systematically. 


Discussion & Conclusion 

Overall, according to current study, anonymity, immediate response, availability and usage in crowded classes are 
the strengths. SMS -based immediate feedback system works without any extra infrastructure and devices anywhere 
and anytime as well. All instructors agreed that SMS -based immediate feedback system is useful and practical for 
crowded classrooms. Similarly, Caldwell (2007) defined response systems as powerful and flexible tools that 
support classroom learning, particularly in crowded classes. The finding of the current study showed that such a 
system has a great potential to involve students into the learning process for lecture sessions. Related pedagogical 
principles and strategies will be further discussed. 

Design issues were encountered as noticeable weaknesses in current study. Based on opinions and suggestions of 
instructors, new version of current system will be redesigned and improved. In special, diversity of graphical 
representation will be increased and web interface of the system will be redesigned to improve usability. 
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Problem 

Design, development, implementation, and evaluation of high-quality instruction for broad dissemination 
involve complex transactions among instructor-experts (subject matter experts), instructional designers, and learners. 
Geographic distance between team members increases that complexity. To work effectively, a diverse, distanced 
group of experts with very different skills and perspectives need to be able to simultaneously teach and learn from 
one another. In the past, a strict division of labor between instructor/experts and instructional designers as well as a 
linear development process imposed by instructional designers were common practice (Berge, 1995). These rigid 
approaches can lead to teams that are unable to conduct processes efficiently and/or generate effective products. 
“Without strong micro-collaboration, the chances of developing a high-quality interactive learning object are slim 
indeed” (Aleckson & Ralston-Berg, 2011, p. 14). 

In an NSF- funded project, a collaborative, interdisciplinary team was formed to design, develop, evaluate, 
and implement an Ethics in Genomics course for genomics undergraduate and graduate students. Advances in 
genomics technology transform society, politics, culture, economies and the environment. Scientists and engineers 
who develop genomics technology need to address key ethical questions about the process of genomics research, 
and the role of society in genomics research. Accordingly, the importance of ethics education for science and 
engineering students is often emphasized, but such educational opportunities often focus narrowly on health, privacy, 
and professional ethics. To address the needs for ethics education in genomics, university professors in Genomics 
and Philosophy formed a team to develop a course that addresses social and ethical issues in genomics. The key goal 
of the course is to facilitate student understanding and ability to think carefully and critically about social and ethical 
questions raised by genomics technology. Given the pressing needs for ethics education in genomics, the 
instructional design team contemplated broad dissemination of a course and decided to develop that course for 
online dissemination. Also, instructional designers proposed a case -based learning approach as an effective way to 
teach the genomics ethics course with e-learning modules. 

Our instructional design team includes: 1) three genomic scientists from three universities, 2) two 
geophysical scientists at one of those universities, 3) a philosophy professor and philosophy research assistant at one 
university, 4) two instructional designers from a university, and 5) an instructional designer from an agricultural 
extension service. The team members are geographically located at distances from each other as they are in three 
different universities in three different cities across two states. Given their diverse backgrounds, the team members 
hold differing views and experiences with college teaching, pedagogical methods, and online teaching. Shortly after 
the team was formed, we recognized the difficulty the team was having with collaborating to develop high quality 
instruction. Difficulties arose from the multidisciplinary perspectives and distances between team members. 
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Solution 


When teams work well together applying their expertise to design and develop instruction, high-quality 
instruction can be the result. In order to facilitate teamwork for instructional design and development of the Ethics in 
Genomics course, we adopted Aleckson and Ralston-Berg’s (2011) micro-collaboration model. Aleckson and 
Ralston-Berg attach the prefix “micro” to “collaboration” to distinguish instructional design and development teams 
from other types of collaborations by claiming that micro-teams work interpersonally rather than interdepartmentally 
or inter-organizationally. The model is meant to promote flexibility and quick change in design features. In the 
model, an egalitarian team communicates regularly in structured activities based on reflection and feedback. The 
micro-collaboration framework consists of five factors to guide teamwork: politics/flat power relationships, 
structure/effective management, culture/shared language, performance/formative evaluation, and momentum, which 
is delineated by the other four factors. 

The first factor, politics, focuses on flattening power relationships, mutual respect, and valued expertise to 
create a high- functioning team. Team members openly share their expertise recognizing that interdisciplinary 
projects often have times when some members are not fully onboard with directions the project is taking. The team 
needs to grapple with the multiple perspectives inherent in collaboration and make efforts to establish quality 
standards and processes for continuous improvement. Learners are at the core of the design and development 
process providing data to inform iterative design and revision. Interdisciplinary teams, including representative users, 
make it a shared goal to create an award winning product. 

The second factor, structured project management, involves a designated project manager who provides 
tools and guidance necessary for the team to be productive. Our team adopted the AGILE management process 
(Sweeney & Cifuentes, 2010) coupled with an ADDIE-like development model for authentic and experiential 
learning (see Figure 1). The team applies the Case, Problem, and Project-based Learning Model for Instructional 
Design to facilitate rapid prototyping. We work from a detailed schedule consisting of goals and objectives, 
deliverables with associated task lists, timelines, including weekly meetings, reporting, and evaluation requirements. 
Our iterative design process is to create an - 

alpha version > deliver and continuously test, build, and revise, 
beta version > freeze and deliver and test > revise, 
pilot version > freeze and deliver and test > revise, and 
final version > disseminate broadly as a MOOC. 

Sharing, discussing, and storing planning documents among teammates creates transparency and leads to 
understanding and trust. 
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Figure 1. Case, problem, and project-based learning model for instructional design of 
authentic and experiential learning 


The third factor, a culture of shared language, further emphasizes the importance of shared documentation 
in effective development projects. The project manager clearly conveys process and product expectations and is 
flexible giving team members a voice in process decisions. Instructional designers communicate the theoretical 
foundations for the pedagogical choices they make and give instructor/experts insight into why they do what they do. 
In addition, they acquaint themselves with the language of the subject-matter experts in order to effectively translate 
that knowledge for e-learning. All meetings have explicit purposes with visual representations of intended outcomes. 
When instructional designers communicate what the deliverables will ultimately look like and offer by showing 
mock-ups or examples, instructor-experts become less resistant to transferring their lecture type materials to digital 
delivery systems. Similarly, instructor-experts need to communicate their tacit knowledge for inclusion in effective 
case, problem, or project-based instruction. Such knowledge is what distinguishes them as experts and includes such 
insights as- what resources they draw upon regularly, how they know that their efforts have been successful, and 
what things a novice-approach might fail to account for? Aleckson and Ralston-Berg (2011) affirm that effective 
team communication depends upon a constant “focus on project objectives and the free flow of ideas and stories” (p. 
102), among team members no matter what their role. 

The fourth factor, performance/ formative evaluation is continuously conducted to include development of 
the e-learning digital components and modules as they are implemented with students. Stufflebeam’s (2002) CIPP 
Evaluation Model, a comprehensive framework for guiding evaluations of programs, projects, personnel, products, 
institutions, and systems is used to evaluate the project. This model’s four components are context, input, process, 
and product evaluation, corresponding to the letters in the acronym CIPP. In general, these four parts of an 
evaluation respectively ask, “What needs to be done? How should it be done? Is it being done? And, did the project 
team succeed?” Emphasis is on evidence-based decision making, objectives, and public relations (Flagg 1990). 
Context evaluation provides a rationale for determining implementation objectives, defines the relevant environment, 
describes the desired and actual conditions pertaining to that environment, identifies unmet needs and unused 
opportunities, and diagnoses the problems that prevent needs from being met and opportunities from being used. 
Input evaluation focuses on resource and strategy decision-making and intended outcomes. Process evaluation 
occurs as the products are being developed and consists of reviewing the products and making decisions concerning 
changes or modifications that are needed. Product evaluation determines the value of project deliverables. In 
addition, Tessmer’s (2001) checklist cited by Aleckson and Ralston-Berg provides guidance for developing a 
detailed evaluation plan that matches the iterative design timeline with the evaluation timeline. Sharing the 
evaluation plan with the team and collaboratively following it pushes the project forward. 
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Carefully addressing the first four factors facilitates project momentum, the fifth factor. Neglecting any of 
the first four factors negatively impacts momentum. Team politics will remain positive as long as everyone’s 
contributions are regarded as essential. Adopting an iterative design process with a designated project manager will 
assure team productivity. While it may seem obvious that effective communication is fundamental to the team’s 
positive impacts, communicating among disparate and geographically separated team members is not easy. Efforts 
must be made on all team members’ parts to acquire and use shared language and to create a common vision of the 
deliverables. An effective instructional development team relies 

“heavily on narrative to accomplish their educational goals. In order for narrative to be effective, as 
opposed to merely entertaining, it needs to be grounded in the real-life experiences of the instructor-expert. 

It is only through such grounding that the instructor-expert’s tacit knowledge can be imparted to the learner” 
(Aleckson & Ralston-Berg, 201 1, p. 129). 

Successful projects involve formative evaluation via frequent user-testing. Feedback from such events 
“amplifies momentum by sparking communication and highlighting forward progress” (Aleckson & Ralston-Berg, 
201 1, p. 130). Authors of the Micro-Collaboration Model suggest the following activities for maintaining 
momentum: benchmarking and theorizing, artwork introduction early in development, prototyping, user-testing and 
reflection, presenting iterations, creating a culture that embraces the continuous revision process, and entering the 
project into contests for awards, conference presentations, and publications. 

We are testing the combination of five micro-collaboration factors as a solution to the difficulty of 
developing instruction among interdisciplinary, geographically distanced team members. The micro-collaboration 
model is derived from both scholarship in academe and experience in instructional development in the private sector. 
The underlying assumption is that creating effective working relationships is a first step toward developing effective 
instruction. 


Research Questions 

The purpose of the study is to examine the effectiveness of the micro-collaboration model in the instructional 
design and development process and to validate the model. The article also provides guidelines on how to 
successfully achieve micro-collaboration between instructional designers and subject matter experts separated by 
geographic distances. Our efforts to examine the effectiveness of the micro-collaboration model were guided by the 
following research questions: 

1 . How satisfied is the design and development team with their product development experience? 

2. How easy is the micro-collaboration model to apply when coupled with a modified ADDIE instructional 
design model? 

3. How conducive/important are each of the five factors of the micro-collaboration model in facilitating 
efficient and effective team processes for application of a modified ADDIE instructional design model? 

4. What are the constraints that impede the implementation of the micro-collaboration model when 
interdisciplinary collaborators work at a distance? 

Testing the Solution - Methods 

The researchers use design and development research as the research method to validate the micro- 
collaboration model and generate guidelines for instructional designers who work with a diverse, distanced group of 
subject matter experts. Richey and Klein (2008) define design and development research as “the systematic study of 
design, development and evaluation processes with the aim of establishing an empirical basis for the creation of 
instructional non-instructional products and tools and new or enhanced models that govern their development” (p. 
748). As indicated in this definition, there are two types of design and development research. The first type relates to 
studies of instructional product development. The other type of design and development research pertains to studies 
of the development, validation, and use of design and development models and processes. This study aligns with this 
second type of design and development research as it focuses on exploring the effectiveness of the micro- 
collaboration model coupled with a modified ADDIE instructional design model. 

Participants 

Participants of the study are the members of an interdisciplinary, instructional development team that 
include five instructor/experts and three instructional designers. Two of the designers are also the researchers of this 
study. One of the instructor-experts is a professor of Genomics at a large public university in the southern United 
States. She is an expert in genomics who conducts genomics research and teaches several genomics courses. 
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Another instructor-expert is a professor of Philosophy at the same university. She is an expert in applied ethics who 
conducts research on issues and problems relating to animal and environmental ethics. Another instructor-expert is a 
doctoral student in Philosophy who studies animal ethics. Two other instructor-experts are geography professors 
specializing in nature-society relations and human-environment interactions. All instructor-experts but one 
geography professor have taught a face-to-face course on genomics ethics, but none of them have taught a fully 
online, blended, or web-enhanced course. 

One of the instructional designers is a professor and director of distance education at a public university in 
the southern United States. She has nearly 30 years of experience as an instructional designer and has designed, 
developed, and taught numerous online courses. She is also an expert in case -based learning. Another instructional 
designer is a post doctoral researcher who holds a doctoral degree in instructional design. She has several 
experiences in designing, developing, and teaching online and in designing and developing online learning 
environments. These two instructional designers are also the researchers who implement surveys and conduct 
interviews with participants. The position of the designer/researcher is comparable to the role of participant observer 
in qualitative research. 

A third instructional designer is a program coordinator in instructional design for the Texas Engineering 
Extension Knowledge Engineering program. She has extensive experience in educational content development, 
evaluation, and assessment. Her role in the project is to coordinate broad dissemination of the course through open 
access. 

Data collection and analyses 

A micro-collaboration model survey is being constructed to assess the degree to which the team was 
satisfied with the collaboration process in the development of a case-based online course, ease of use of the model, 
the importance of each of the five factors for effective team process, the degree to which the instructional 
development team successfully implemented the model, and constraints to micro- collaboration. Participants will 
complete the survey twice: in the midst of developing the alpha version of the course and toward the end of 
developing the beta version. With a small number of study participants, the survey data will be analyzed using 
descriptive statistics. 

Three additional data sources will be collected to complement the survey data: design journal, design 
documents, and interviews. One of the instructional designers keeps a journal that includes critical incidents and 
lessons learned during design and development processes. Documentation is constructed roughly every week. 

Design documents involve a continuous record of design issues, tasks to be done, responsible parties, task due dates, 
and task status. Lastly, interviews will be conducted with instructor-experts and the instructional designer in charge 
of dissemination to explore in greater depth their experiences of the collaboration process. The interview protocol 
includes open-ended questions that prompt the five factors of the micro-collaboration model as well as other aspects 
of the team process (e.g., difficulties and challenges experienced in the development process). Interviews will be 
audio-recorded and transcribed verbatim. The analyses of the design journal, design documents, and interviews will 
include open-coding and theme generation (Fereday & Muir-Cochrane, 2006; Strauss & Corbin, 1990). Data 
collection is in progress. In the following section, we will share the preliminary findings. 

Preliminary Findings 


Team satisfaction 

Thus far, most of the team works well together, meets regularly, and delivers products as scheduled. 
However, one of the instructor-experts with essential content expertise at one of the universities, is quite illusive. 
Other team members are complaining that she cannot be reached, will not respond to attempts at contact, and is not 
making substantive contributions as a result. Being at a distance makes it difficult to intervene and assure that she 
connects with and contributes to the project. Without that connection, micro-collaboration strategies will be 
ineffective. As it is early in the project, the team will continue to use strategies in the model to help the group 
collaborate effectively. 

The instructional designers value the model perceiving that it has provided practical suggestions that have 
proved helpful and has served as prompts for processes to apply. Instructor-experts appear to appreciate our efforts 
to organize project activities and appear to be motivated by products that we provide for them. 

Ease of use of the micro-collaboration model 

The instructional designers find the model to be extremely natural as it corresponds to typical instructional 
design processes and adds value to those processes by providing specific guidance for project management. The 
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model’s contribution is that it clearly spells out effective course management tasks, keeping the instructional 
designers mindful of how to respect team members and move forward productively. 

Impact of each factor in the micro-collaboration model on use of the instructional design model 

Politics/Flat power relationships 

Aleckson and Ralston-Berg (2011) suggest that effective communication of instructional designers’ 
expertise and instructor-experts’ demonstrations of passion for content helps build trust among the team members. 
They emphasize the importance of equal power relationships between instructional designers and instructor-experts. 
In our team, instructional designers and instructor-experts have worked together to create an overall design plan and 
determine course objectives and activities. During this process, instructional designers have articulated rationales for 
developing particular learning materials (e.g., cases and rubrics) and used the instructor-expert’s vocabulary when 
possible while expressing respect for expertise of the instructor-experts. For example, when prototyping the 
structure of each module of the course, both instructional designers and SMEs proposed their respective prototype. 
Then, the instructional designers combined elements of the two proposed prototypes by reflecting on the ideas of the 
instructor-experts as much as possible and incorporating their vocabulary. By doing so, the instructor-experts have 
learned to trust the instructional designers, and such trust among the team members promotes effective collaboration 
over a distance. 

Effective management 

Instructional designers created a project management website that is shared among the team members. This 
website contains the meeting logs, the list of deliverables, timelines, tasks to be done, task status, and other design 
documents. The shared documentation allows the project managers to effectively communicate process and product 
expectations over the distance so that every member of the team is clearly aware of the progress and status of the 
project and knows what he or she needs to do next. As well as guiding the formative evaluation process, Tessmer’s 
(2001) checklist guides project management and facilitates teamwork around formative evaluation tasks. 

Performance/Formative evaluation 

Planning for the iterative processes of formative evaluation helps the team move forward. The deliverables 
associated with formative evaluation processes enable the entire team to see the proximal goals and collaborate 
effectively toward meeting these goals. Our plan to create four iterations of the course and evaluate each prototype 
with users follows the CIPP model and Tessmer’s checklist. Context evaluation in this project is addressed through 
means such as the following: 

• Identify additional graduate students and programs that will benefit from the addition of this ethics 
education course to the program curriculum. 

• Establish partnerships with specific programs and academic departments for adoption of the Genomics & 
Society course. 

• Establish contacts with eminent ethics and genomics experts to obtain their perspective on priorities for 
integrating ethics education into the graduate science curriculum; seek and gain their initial input, and 
participation in the class modules. 

• Determine how to best distribute the e-learning modules beyond the 3 initial partners. 

• Identify the most appropriate student subpopulation (sampling procedure) for testing the curriculum. 

Input evaluation is met through the following activities: 

• Project staff meet weekly to evaluate project progress and determine modifications or alternative strategies 
needed. 

• Assess instructor and student use of the Genomics & Society course curriculum in order to make 
modifications. During the semester, assessment will occur after each module. 

• Conduct follow-up of students who experience Genomics & Society curriculum and use information for 
input on further program development. 

Process evaluation is met through the following activities: 

• Collect and analyze learner data through the following methods: learner demographics to assure successful 
delivery to underrepresented populations, usability testing, pre- and post-evaluation of exercises, embedded 
tests, questionnaires, interviews, and focus groups. 

• Conduct debriefings to improve the ways in which team members work together. 

• Consult among project staff and faculty on the perceived usability of the Genomics & Society ethics 
curriculum. 
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The success of the product offerings is determined through the use of pre- and post-testing to identify quantitatively 
the extent to which there is an improvement in critical thinking skills, and an expansion of the context in which 
students consider the potential implications of genomics and technological advancements. The use of participant 
feedback forms provides a qualitative method for the participants to evaluate their appreciation of the course 
activities, the usefulness of the course, and the quality of the instruction. A survey is conducted 6 months after 
training by contacting at least 80% of the participants to evaluate their incorporation and application of their 
acquired knowledge to their specific research or work field. 

Culture/Shared language 

Establishing a culture of shared language promotes our instructional design and development processes. 
Whereas the instructional designers recognize the applicability of case-based learning (CBL), our instructor-experts 
had never taught using this instructional approach and had many questions about how to teach this way. 

Instructional designers thus spend a significant amount of time in communicating the methods for teaching using a 
case-based learning approach (Jonassen, 2004; Naumes & Naumes, 2006) so that all the team members embrace its 
value and establish the common language. With the shared understanding of case -based learning, the team is able to 
maintain effective communication that advances the design and development process. For example, the team 
successfully collaborates on developing cases and rubrics as they revise. Also, as suggested by Aleckson and 
Ralston-Berg (2011), instructional designers share sample rubrics and cases, and this seems to alleviate the 
discomfort that the instructor-experts have experienced making a transition to the new instructional approach. 

In addition, the instructor-experts have never taught a course online prior to this project and currently do 
not buy-in to the possible effectiveness of online courses. Currently, they plan to teach the course face-to-face 
throughout the project and beyond and only develop the online course for national dissemination beyond the three 
participating universities. While instructor-experts intend to deliver online components during the project solely for 
purposes of formative evaluation, instructional designers envision ultimately delivering iterations of the course fully 
online during the development process. The instructional designers hope that by regularly sharing the online course 
as it develops, instructor-experts may ultimately appreciate the potential effectiveness of the fully online course and 
be willing to adopt it for delivery from their universities. 

Momentum/Political, structural, cultural, and performance 

Having weekly meetings and sharing planning documents weekly have most positively impacts team 
momentum. For instance, instructional designers share the ADDIE-like Case, Problem, and Project-based Learning 
Model for Instructional Design to instructor-experts who then understood the design and development approach 
taken by the instructional designers and enthusiastically contributed criteria for assessment rubrics as well as rich 
cases to be presented in course module. Instructional designers also shared example cases, and rubrics to facilitate 
instructor-experts’ building of those for course content. Sharing project management documents with tasks, 
benchmarks, and person responsible for each task has kept those on the team who attend meetings productive. The 
entire team shares access to the Blackboard course environment and can see it as it grows. This contributes greatly 
to momentum. 

Constraints 

We recognize several constraints that hinder effective, efficient micro-collaboration. One was insufficient 
opportunities for informal interaction due to distances between team members. According to the model maintaining 
social relationships is necessary for building trust and bonds among team members. Especially for the group of 
people with different skills and perspectives, bonding time allows team members to understand differences and build 
common ground for collaborative communication. However, because team members were geographically located at 
distances, we rarely have an opportunity to socially interact and get to know each other. 

In addition to lack of social bonding, the team encounters difficulty scheduling regular meetings that all 
members can attend. According to Aleckson and Ralston-Berg (2011), regularly scheduled meetings are imperative 
for the team to make steady progress and promote effective, efficient communication. Because our design team 
consists of professionals from various fields and the team members are geographically located at distances from 
each other, it is challenging but imperative to schedule regular, frequent meetings. 

Also, because the instructor-experts do not have substantial experience with online, blended, or web- 
enhanced courses they do not necessarily see themselves teaching the course that we are developing. Instructional 
designers find that the experts’ lower commitment to teaching the course online, impacts collaboration in the overall 
instructional design processes. 
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Another constraint relates to the instructor-experts’ unfamiliarity with instructional design processes. In the 
Case, Problem, and Project-based Learning Model for Instructional Design, as in other ADDIE models, a first step 
in the instructional design process is to conduct goal analyses. It seemed to be challenging for the instructor-experts 
to identify performance-based goals and objectives. Therefore, the instructional designers began the goal analysis 
process by collaboratively developing rubrics for evaluating students’ case analyses. The rubrics served as a way for 
the instructor-experts to describe desired performances and specific criteria. As the instructor-experts identified 
evaluation criteria in the rubric, the instructional designers were able to extract the appropriate instructional goals. 

Furthermore, we found some tension between instructional designers and instructor-experts with regard to 
the degree of structure in the course development. Although it is typically recommended to have a consistent 
structure across different units within an online course (Clark & Meyer, 2011), we found that the instructor/experts 
wanted to have more flexibility within units. Thus, the instructional designers had to find a balance between 
consistency and flexibility. 


Guidelines 

1 . List concrete action items of the micro-collaboration model, share them with the entire team, and regularly 
check that each member of the team is on board with processes. 

2. In order to establish trust among the team members, the team needs to invest in building and maintaining 
personal as well as professional relationships. 

3. Create a shared project management website. 

4. Frequently develop and share iterative documents. Instructional designers need to continuously develop and 
show initial rough Web-based prototypes to illustrate as close an approximation as possible to the functionality 
and potential of the online course. 

5. Create working documents on a shared drive for collaborative viewing and/or contribution. 

6. Consider conducting your goal analysis in the context of rubric development when working with SMEs who are 
not familiar with instructional design processes. 
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Abstract 

This research examines a number of academic communities with the intention of expanding the sphere of research 
across academic and career disciplines for adult learners. Most electronic communities have little crossover 
communication with each other beyond specific linkages to other sites. So there is scant crossover among the world 
of academia, the “world of work,” and the virtual world of social media. The objective of this research is to examine 
e-learning communities across academic disciplines to explore “crossover” research in multiple academic disciplines 
to see if there could be key issues brought to the foreground that affect adult learners and student retention. In 
addition to conducting qualitative historic research across a number of academic disciplines, the study also provides 
a case study that could reveal insights that shed more light on adult learners and best practices on retention in online 
courses. Traversing academic disciplines can uncover key factors that could contribute to the body of research on 
online adult learners, including student retention. 


Introduction 

At the beginning of each research project, it is likely that the seminal studies researched on that subject will 
come from the academic discipline of the researcher. It is also likely that the entire literature review will be 
conducted largely in that researcher’s academic field, with infrequent exceptions. Due to the mass availability of 
global electronic databases, however, researchers can now stumble upon expert research studies from other 
academic disciplines that they were not aware of before and which could shed new light on old questions. 

What does this have to do with adult learners and student retention? While most universities try to budget 
monies and resources for new technologies to accommodate social media, few are looking at ways to restructure 
internally to be more open to interdisciplinary collaborations among their colleges. University plans may provide for 
overall budgetary needs and university goals but may also need to focus more on how paths can be created across 
disciplines to both contribute to research, and more important, to help students learn and complete online courses. It 
is important to find ways to combine all the research in as many disciplines as possible to support best practices of 
how students learn new materials and stay engaged enough to accommodate new content knowledge with new 
technologies and with what they already know. 

A factor that seems to be eluded in the rush to create online courses and programs to attract students is what 
they already know when they come to that course or college. There is ample research available on adult learners and 
how they learn (Cassara, 1990; Daloz, 1999; Elias & Merriam, 1995; Knowles, 1973; Lindeman, 1926; Merriam, 
1995; Merriam & Brockett, 1997; Merriam & Caffarella, 1999; Sabatini, Daniels, Ginsburg, Limeul, Russell, & 
Stites, 2000). Adult learners want results; they want to know that what they are learning contributes to their 
knowledge and helps them to meet their goals. Making sure that prior knowledge is considered in online learning for 
adults is important, particularly because, as of 2013, students are older than traditional age students of the average 
ages of 1 8-22, and they come to college with a vast array of experiences. While traditional age students still 
predominate at four year universities, there are a growing number of students who are generally older, averaging 23- 
27 years of age and older. About 43 percent of undergraduates enrolled in postsecondary education in the United 
States are age 24 or older (National Center for Education Statistics, 2007, 2011). At least 36 percent of 
postsecondary students were age 25 or older, and 47 percent were independent students (Center for Postsecondary 
and Economic Success, 2011; Hudson, & Shafer, 2005; National Center for Educational Statistics, 2007, 2008a, 
2010, 2011). These students more often fall into the definition of what used to be called nontraditional students who 
often have responsibilities, such as work and families, so they are charged with wanting to get their degrees 
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expeditiously and to take courses that will help them progress towards their careers. Accordingly, because of the 
additional responsibilities and time requirements, these students often choose online courses. 

But the course completion rates and graduation rates of students enrolled in online courses are much lower, 
at best, averaging 60% completion rates (Gallop, 2013), so, the growing question with cause for concern is, with the 
challenge of more students being attracted to online courses, how can administrators help students to have higher 
completion rates in those online courses and programs? 

Exploring Interdisciplinary Research 

The researcher in this study wanted to pull together all of the possible ways to discover best practices 
across academic disciplines that demonstrated success in online student retention. She first researched the disciplines 
of Computer Science and Software Engineering, Education, Commerce, and Human Sciences, and Curriculum and 
Instruction on eLearning, paying attention to any research available on distance and e-learning in those disciplines. 
She then found there was also a growing body of research on distance learning as a separate field of research, which 
she then explored in order to uncover seminal studies in that field to integrate them with those of the other academic 
disciplines. She was also careful to search for research related to successful online course completion and student 
retention in each of those disciplines as well as for any interdisciplinary studies related to elearning. 

Computer Science and Software Engineering 

In the body of research available in Computer Science and Software Engineering related to eLearning, 
several seminal authors emerged (Kinshuk, Suhonen, Sutinen & Goh, 2003; Kinshuk, Patel, Oppermann, Russell, 
2001; Patel, Kishuk & Russell, 2003; Kinshuk & Patel, 1996; Kinshuk & Yang, 2003; Tretiakov, Kinshuk & 
Tretiakov, 2003; Lanier, 2004, 2012) and these studies largely dealt with applications, data mining, instructional 
system designs, learning objects, and game designs (Elias & Merriam, 1995; Gee, 2007; Ifenthaler, Isaias, Spector, 
Kinshuk, Sampson, 2011; Giarratano, Gannon Cook, 2001; Kinshuk & Patel, 1996; Kinishuk & Yang, 2003; Lanier, 
2004, 2011, 2012; Yang, 2006). 

Education, Curriculum and Instruction 

There is also an extensive body of research available in the academic disciplines of Education, Curriculum 
and Instruction, adult learning, (Anderson, 1998; Baumgartner, Seethaler, Cheng, Lo, & Slotta, 2000; Berker, & 
Horn, 2005; Bonk& Cunningham, 1998; Bradburn, Berger, Li, Peter, & Rooney, 2004; Brockman, 2005; Brophy, 
1968; Brown, 2001; Cassara, 1990; Crawford, Gannon Cook, Varagoor, 1998; Clark, Herter, & Moss, 1998; 
Cunningham, 1992; Daloz, 1999; Darling-Hammond, 2006; Driscoll, 2001; Elias & Merriam, 1995; Gannon-Cook, 
2012; Hord, Jolly, Mendez-Morse, & Vaughan, 1993; Hudson, & Shafer, 2005; Lave & Wenger, 1991; Lepper, 

1988; Lindeman, 1926; Merriam, 1995; Merriam & Brockett, 1997; Merriam & Caffarella, 1999; Molebash, 2002; 
Robinson, 1996; Sabatini, Daniels, Ginsburg, Limeul, Russell, & Stites, 2000; Sergiovanni, 1994; Jonassen, 1986, 
1997). 

In addition, there has been a growing number of adult learners returning to complete their education too; 
the researcher sought to examine studies on adult learners to see what the data revealed on enrollments and 
retention, particularly with respect to online learning (Allen & Seaman, 2010; Berker & Horn, 2005; Bonk, Kirkley, 
Hara, & Dennen, 2001; Bonnema & Van der Waldt, 2008; Boston, Ice, Gibson, 2011; Chambers, 2004; Davis, 2001; 
Lindeman, 1926; National Center for Educational Statistics, 2007, 2008a, 2008b, 2010, 2011; Raisman, 2010, 2011; 
Sabatini, Daniels, Ginsburg, Limeul, Russell, & Stites, 2000). 

Commerce 

While there is a wide and extensive body of research in the field of commerce, the researcher focused on 
the aspects of business that related to training, motivation, project management and marketing (Covey, 1990; 
Csikszentmihalyi, 1990; Deming, 1993; Gallop, 2013; Gelb, 1998; Herzberg, 2001, 1987, 1964; Kaufman, 1987, 
1992, 1998, 2000: Kaufman & Swart, 1995; Kaufman & Watkins, 1996; Kaufman & Watkins, 2000; Kaufman, 
Thigarajan, & MacGillis, 1997; Kaufman, Watkins, Triner, & Stith, 1998; Naisbitt, 1975; Pink, 1996; Zaltman & 
Zaltman, 2009). She also looked for research and seminal articles that related to online learning. 

Human Sciences 

In addition, the Human Sciences offered opportunities for research on the subjects of psychology, 
sociology, anthropology, linguistics, and semiotics; once again the researcher focused on seminal studies related to 
aspects of teaching and eLearning (Baumgartner, Seethaler, Cheng, Lo, & Slotta, 2000; Davis, 2001; Driscoll, 2000, 
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2001; Eco, 1976, 1986, 1988; Fullan, 1984, Gannon-Cook, 2011; Lanier, 2004, 2011,2012; Lantolf & Thome, 

2006; Parker, 2004; Penrose, 1990; Rogers, 1986; Schlain, 1998). Anthropology and philosophy have some 
crossover topics with other disciplines in the humanities, (Levi- Strauss, 1958; Peirce, 1988), such as semiotics, and 
many of the disciplines within the Humanities are now also being taught via hybrid and online courses; so seminal 
studies in these disciplines could also contribute to interdisciplinary and distance education research. 

Distance Education 

Over the last fifteen years the emerging field of Distance Education has emerged as a discipline with its 
own library of research. There has been seminal research conducted, but there is necessarily crossover with other 
topics, such as the academic areas of studies mentioned here (Bednar, Cunningham, Brockman, 2005; Berge & 
Muilenburg, 2001; Duffy & Perry, 1995; Berge & Muilenberg 2001; Bonk & Cunningham, 1998; Davis, 2001; 
Gallop, 2013; Gannon-Cook, 2006; Kinshuk & Yang, 2003; Mager 1997; National Center for Education Statistics, 
1995, 96, 2007, 2008; Portugal, 2006; Reigeluth, 2001; Seels & Glasgow, 2001). 

Y et there are important research gaps in all of the above-mentioned fields of study, particularly with respect 
to eLearning and social media. Perhaps some reasons for the gaps could reside, at least partially, from the 
segmentation of all the research in these disciplines; ironically, none of these are separated and restricted from 
perusal when a person conducts an Internet Google or Wikipedia search on a topic. In fact, that is exactly what most 
students do — they will search for information by topic, not by academic discipline. 

Unearthing the Social and Cultural in Virtual Realms 

A group of luminaries as varied as the topics of their expertise, such as J. Doyne Farmer, a physicist with 
the Santa Fe Institute, shared his belief that experts should be more well-rounded. “I knew that real physicists 
weren't just supposed to know about physics; they were supposed to be broadly educated and know a little (or, better 
yet, a lot) about everything” (Lanier, 2004, available online). Leonard Schlain, a neurosurgeon by profession, sees 
fit to digress from medicine to write about ancient connections between vision, oralcy and written communication 
(Schlain, 1998). Umberto Eco, Chair of the Department of Semiotics at the University of Bologna, Italy, also writes 
about the social and cultural influences of history, and writes historical novels. (In addition, he critiques experts on 
the differences between methodological and ontological conceptions of structure and structural anthropology to 
structures in reality (Parker, 2004). Other “seekers” who venture beyond their academic purvey include Roger 
Penrose (1995), who wrote the” Emperor’s New Mind” that included a series of observations about Mathematics 
and Physics; he extended musings beyond those topics into the realms of science that asked questions about the 
universe’s origins that he debated with colleagues like Steven Hawkings. Albert Einstein not only presented his 
seminal theory of relativity, but shared his enthusiasm for both science and music. Contemporary electronic 
visionary Jaron Lanier of Stanford University coined the term “virtual reality” and is known for his pioneering in 
computer science arenas, but extends his interests beyond computers and software to writing symphonies and 
creating electronic art (Lanier, 2004). 

The cast of academics who share their love for research across many disciplines goes on, hopefully 
perpetuated by the newest luminaries that care enough to extend knowledge beyond their own disciplinary 
boundaries. Each of these people had a vision, to share thoughts and ideas across the traditional lines of their 
academic boundaries. With so much more data available electronically than ever before, the questions of how to 
make that data meaningful across academic and career disciplines becomes even more paramount. In the electronic 
world, there are no separations or exclusions by discipline, or by perceptions. 

So, how can the process of folding these diverse communities into a larger electronic collaboration of 
eLearning be approached? Is it possible to have dialogues with members of divergent academic communities in a 
broad spectrum of electronic data environments? It seems that too often academics are hesitant initially to approach 
colleagues from other academic disciplines. They may find it difficult to take the time to stop and talk with other 
academic colleagues, even when their colleges reside in the same building. But when students want to find answers, 
their searches almost always begin by going online, and then can vary by searches and browsers, to sites, such as 
Wikipedia, Facebook, blogs (weblogs) and perhaps library sites. While students may be comfortable texting, 
sending photos, tweeting, frustration can build when students feel they are getting overwhelmed with too many tasks 
that they may not feel are relevant which is what often occurs in tasks associated with online courses, especially for 
students who may be taking online courses for the first time. The research supports that, while students are usually 
familiar with texting and cell phones, they are often less confident in their online navigation and assimilation of all 
of the course materials when taking online courses; even students who spend hours daily texting friends (Sutton, 
Gannon-Cook, 2011) voice confusion and frustration over participation in their online courses. So the question 
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remained, what would further enlist students and keep them engaged enough to participate, learn the course 
materials, and persist to complete their courses? 

In studies of adult learners in online courses, factors, such as work and family responsibilities, financial 
stresses, and time management, were cited as accounting for some undeterminable amounts of attrition in online 
courses and programs, but the researcher felt there was more to the story (Patterson, & McFadden, 2009;). Student 
attrition in online courses could also be attributable to factors like cognitive overload, and sheer lack of students’ 
ability to link new online content materials to their prior knowledge. 

Cognitive load and elearners. 

Cognitive Load Theory (Sweller, 1988; 1994) is defined as 

an instructional theory. . . (that) describes learning structures in terms of 
an information processing system involving long term memory, which 
effectively stores all of our knowledge and skills on a more-or-less 
permanent basis and working memory, which performs the intellectual 
tasks associated with consciousness. Information may only be stored 
in long term memory after first being attended to, and processed by, 
working memory. Working memory, however, is extremely limited in 
both capacity and duration. . .the fundamental tenet of cognitive load 
theory is that the quality of instructional design will be raised if greater 
consideration is given to the role and limitations, of working memory. 

(Cooper, 1998, p.l) 

The term cognitive load was first used by John Sweller (1988, 1994) to describe the amount of pressure related to 
the mind and its working memory (WM). Sweller also pointed out that differences in learning (and performance) 
could be due to higher levels of cognitive load; the more stress, activities, and information added to a person’s short 
term memory, the more that person has difficulty processing and retaining the information. There are varying 
opinions as to how much information becomes too much before information overload, but there seems to be a 
general consensus that around seven to nine pieces or “chunks” of information can be retained before a mind begins 
to feel overloaded and begins to experience greater stress (Chandler, & Sweller, 1991; Chang, 2006; Gannon-Cook, 
2006, 2011; Miller, 1956; Sweller, 1988, 1994). 

Cognitive overload is often difficult to identify in students, particularly online students because most online 
courses are asynchronous with no visual encounters; students are usually monitored solely by the instructor 
assessing their homework assignments, evaluating their emails and interactions. Ultimately, the success of the 
students is determined by the students’ course assessments and by their successful (or unsuccessful) completion of 
the course. 

What occurs in online courses is that students need to learn the new course materials, but they also have to 
already be familiar with, or, if not, need to learn, how to use the technology and become more comfortable with 
using it in order to navigate and participate in the course effectively. (See Figure 1). 


Figure ; 1 

Steps required to effectively navigate an online course. 



The steps involved in participating in an online course require at least eight actions, 

1. getting to the site, 2. entering the passwords, 3. looking at all the links and information on the course homepage, 
4. then selecting the content modules or discussion conferences, 5. going to the checklist and assignments page, 6. 
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perhaps leaving the site to do the assignment (or going to the email link to email the instructor with questions), 7. 
returning to the website and going through the first four steps again, and 8. Posting the assignment or discussion in 
the appropriate week module. There may be other activities also required, such as navigating to links in the course 
module, or perhaps assignments that require timed synchronous chats, or drop boxes that must be used to submit 
assignments, all of which require additional navigation or interaction. If, according to the theory of cognitive 
overload, seven to nine chunks of information or activity can cause a mind begins to experience overload, then it 
begins to make sense that prolonged and ongoing confrontations with all of these required activities could confront 
and discourage even tech savvy students in online courses. 

Processes related to higher-order thinking, such as problem-solving and knowledge transfer, can lead to a 
multiplier effect that exacerbate the effects of all these concurrent activities on learners and result in increased stress 
or anxiety (Baddeley, 1992; Braxton, 2000, 2004; Center for the Study of College Student Retention, 2009; Cooper, 
1998; Galbraith, 2004; Jaschik, 2010; Patterson & McFadden, 2009; Pavio, 1990; Tinto, 1993; U.S. Department of 
Education, 2009). Add the cognitive load factors that come from participating in online course activities with daily 
life challenges and the pressure increases exponentially, so it becomes a burden that can be overwhelming to 
students in these online courses. Considering all of the tests required to persisting through to completion, it is not 
surprising that attrition rates are higher in online courses. Research has been conducted on activities, such as 
collaborative groups, social networking, and authentic fast instructor responses, that can encourage student 
participation and interactivity in online courses that have reported benefits to the inclusion of these activities (Allen 
& Seaman, 2010; Boston, Ice, Gibson, 2011; Gannon-Cook 201 1; Patterson & McFadden, 2009; Sutton, Gannon- 
Cook, 2012). 

But there should be an ongoing desire to investigate what could help students complete online courses, so 
there is a need to take into consideration not only the factors that have already been reported to contribute to course 
completions, and the need to continue to incorporate new technologies and content materials mandated by 
sponsoring universities, but to look for other factors that could meaningfully contribute to student retention and 
completion of online courses. 

Strategic planning of how to simplify the linking of new knowledge to prior knowledge is seldom 
considered in the design of online courses. The researcher sought to look for studies that researched how students 
could enter online courses feeling more comfortable about taking on the requirements of online courses. She thought 
that if there was a way to link online courses to what students already know it could reduce their anxiety at the 
beginning of the courses and, therefore, ease their apprehension about the cognitive burdens as they proceed through 
the courses. She thought that if historic and sociocultural references were included in the course site portal and 
introductions, students would recognize these, even if subliminally, and these comforting factors may contribute to 
relaxing some of their concerns about participating in the online course. 

Vygotsky (1978, 1985, 1986) offered the basis for a culturally grounded theory of cognition, with the 
concept of “mediated tools” linking culture to the functions of consciousness. Tool mediation is “the process 
through which humans deploy culturally constructed artifacts, concepts and activities to regulate (that is, gain 
voluntary control over and transform) the material world of their own and each other’s social and mental activity” 
(Lantolf & Thorne, 2006, p. 79). Linguist Ferdinand de Saussure and philosopher Charles Peirce disagreed on some 
principles, but both posed that meaning was a cultural construction (Cobley & Jansz, 2004); while Saussure stressed 
the relationship between the signifier (the material conveyer of the sign, primarily a verbal message) and the 
signified (the message recipient), Peirce (Eco, 1976, 1997; Danesi, 2007; Mounce, 1997; Oswald, 2012; Peirce, 
1988); differed by purporting there were three significations that constituted meaning, the signifier (representamen), 
the signified (object) and the interpretant (the result of the encounter with the sign). These signs “trigger a chain of 
associations which eventually seem quite removed from the initial sign” (Cobley & Jansz, 2004, p.26). 

Signs and icons have been found in caves dating back from 20,000 years ago and seem to convey meanings 
that free the sign (signifier) from its literal meaning; while drawings, pictures and stories are legacies from our 
ancestors, they are also social and cultural conveyors of knowledge that are bound through time to the our lives and 
values (Oswald, 2012). These codes “set up a ‘cultural’ world which. . .is linked to a cultural order, which is the way 
in which a society thinks [and] speaks” (p. 11). Cultural myths and archetypes have been passed on to emerge in each 
generation over millennia and their codes are embedded in humans’ collective unconscious. “Codes determine the 
guidelines for interpreting signs” (p.29). Signs and icons are not just pictures to be viewed, but codes that convey 
meaning residing in that vast pool of historic unconscious knowledge and they form the foundations of our language 
and communications. “Semiotic analysis stakes out the broad tensions that structure these myths and identifies ways 
. . .these tensions (can be resolved) by means of strategic positioning. . .and creative strategy. . .myths structure culture 
into constructive units. . .they also structure the meaning of. . .all kinds of . . .texts and rituals” (p. 38). 
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Symbols and codes are used daily to enlist consumers’ nostalgic and subconscious responses to buy 
products; in fact, marketing semiotics provide the advertising industry with the data needed to assess and predict 
consumer buying patterns (Levy, 1959, 1981; Shaik, 2005; Zaltman, 2009). If the inclusion of semiotic 
representation can links prior knowledge to new learning materials, there could be a benefit to following the lead of 
industry to strategically link semiotics to students’ existing knowledge and help them to mediate these tools with 
new online course content. 

In the case of online learning, instructional designers could strategically help students to relax somewhat by 
embedding these icons and metaphors that students recognize on a subconscious level at the start of each online 
course. These comfort signals could help students to be introduced to the course and feel less confused, effectively 
serving as signposts to help them adopt course tools and content materials. 

As semiotics become more commonly integrated into online courses, standards could be established that 
would help instructional designers reinforce learning receptivity patterns; they could also enlist students’ feedback 
on an ongoing basis, to reinforce buy-in from students and bridges to new learning. In research studies of online 
courses using semiotic tools designed to link students’ new knowledge with prior learning (Gannon Cook, 2012, 
Reed, 2012), students remarked in their course reflections that, while the courses seemed similar to others they had 
taken, somehow these courses were different. They seemed to feel more comfortable in them, and, moreover, the 
findings demonstrated that completion rates were higher in these courses (by over twenty percent). 

Bridging Student and Interdisciplinary’ Knowledge 

There are threads of research that cross many academic disciplines, strategic knowledge bridging can not 
only work to help make students more comfortable in online courses, but can also work for adventurous researchers 
who venture across academic disciplines to explore deeper interdisciplinary research and collaborations (Bonk & 
Cunningham, 1998; Bonk, Kirkley, Hara, & Dennen, 2001; Clark, Herter, & Moss, 1998; Crawford, Gannon Cook, 
Varagoor, 1998; Havernik, Messerschmitt, Vandrick, 1997; Hord, Jolly, Mendez-Morse, & Vaughan, 1993; 
Molebash, 2002). The paths to charting new data terrains begin with searches for ways to initiate collaborations 
with adventurous researchers who also want to chart new research pathways. Interestingly, in the 21 st century, most 
academic disciplines have already charted some overlapping territories, at the minimum with computer science and 
online courses. 

Case study examples. 

The researcher had conducted prior collaborative research with a colleague in the College of Computer 
Science, so she hoped to replicate that research at her new university. That research utilized a case study that began 
with a chance greeting at a water cooler with a colleague from a different college at a southwestern university in the 
United States. The project began with informal emails on a mutual topic of interest: student capstone projects. Their 
initial conversations produced a dialogue about the problems students experienced with their graduate capstone 
projects. They discovered that students in both programs were having trouble working in cross functional teams; 
there was particular difficulty in communicating and teambuilding. It appeared that, while students were skilled in 
the knowledge they derived from their academic programs, they had not learned how to use those skills in real world 
scenarios with others. So what resulted was a collaboration between the two departments, each faculty created a 
collaborative interdisciplinary capstone project designed to help immerse graduating computer information systems 
(CIS) and instructional systems design (ISD) students in real-world programming and project management 
scenarios. Most of the students in both departments were international, so there was an opportunity for these 
students to be introduced to local as well as simulated team scenarios. The project helped students develop 
collaborative skills working in teams with professionals from diverse backgrounds and cultures. A “byproduct” of 
the project was their exposure to instructional designs and projects which provided students with a broader 
understanding of how their expertise fit into team collaborations. The result was that these students had a better 
opportunity to convey their understanding of cross functional teams to prospective employers, and, therefore, a 
greater chance for success in getting hired than before they had experienced their capstone course projects (Gannon- 
Cook, 2006). 

The researcher wanted to replicate the study at another university, this time in a slightly different university 
culture where she attempted to again find faculty in other departments who would be willing to offer a similar 
capstone project for graduate students. However, because of a lack of cooperation among departments, she was 
unable to initiate a collaborative capstone project at that university. 

While the data from the original study was only be used as archival data in this one, it provided some 
findings that the researcher used to design a course where the students would work on a capstone project and report 
their findings back to the class. In this case she also included extensive graphics and semiotic tools to cross the 
multi-national and multi-disciplinary borders of the students in the classes and provide a subtle translinguistic 
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semiotic context. The same assignment was given to each group at the start of the quarter with the encouragement to 
use creative license and to work collaboratively. At the inception themes, like pictures of people sitting by a 
campfire telling stories, seemed to help students to be more comfortable communicating with each other initially; 
testimonials were encouraged from colleagues in each group as the course progressed, and these reinforced team 
interactions and helped team functionality (Zaltman, 2009). The collaborative interactions helped students from 
different environments to come together and work toward a common goal. When the time came to turn in the team’s 
projects, the participants had a better understanding of the importance of communication and teamwork in addition 
to simply having content knowledge. 

Lessons Learned 

So, from the first example of the casual “water cooler” conversations, academics from different academic 
disciplines came together to know each other and each other’s research better. The case study with the students also 
helped students have a better understanding of the importance of both working collaboratively and cross 
functionally. Now the hope would be to continue to expand the opportunities for both interdisciplinary and 
collaborative research. 

Building a learning community could also begin similarly by searching for research topics that may reside 
across the line of other academic disciplines. Initial communications and follow up informal emails or Skype 
discussions can bring diverse disciplines together. There are ways to have interactions and dialogs across 
disciplines; communication requires “an artful mix of direction and facilitation requiring more than just technology” 
(Kinshuk, 2003, December 16). 

In university environments it can be difficult to initiate interdisciplinary conversations and collaborations, 
so, depending on the openness of the university environment, it may need to begin with the conversation with 
department heads and, when there is support by supervisors, it is also likely that there will be action in the direction 
of the desired goal. (A number of research studies indicate that change is most often successfully demonstrated 
when it is endorsed by management (Gannon Cook, 2003; Bonk, Kirkley, Hannan & Denna, 2001; Robinson, 1995). 

When considering conducting research in other academic disciplines, the rewards of venturing across 
academic borders to explore other disciplines, the discovery that each and every discipline has an extensive body of 
research that could be mined for important data that could provide exciting new discoveries for interdisciplinary 
explorers in a rapidly evolving global community of learners. 

Discussion 

Investigating and collaborating in interdisciplinary approaches across university colleges and departments 
could introduce multiple approaches and open vast databases of knowledge currently invisible beyond the borders of 
respective academic discipline “categories.” Academics who elect to traverse broader intellectual plains find these 
new “category” borders readily available to crossover simply by conducting an Internet search; those who elect to 
span multiple academic and cultural disciplines can find vast new territories of knowledge bases, like treasure 
troves, waiting to be discovered. 
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Abstract 

This paper describes the development of CATS or Cognitive Affordances of Technologies Scale, a tool that 
can aid faculty, instructional designers, and e-learning specialists in identifying and leveraging the cognitive 
affordances of a learning technology. CATS includes seven multi-disciplinary research supported categories of 
cognitive affordances that can be harnessed to enrich student learning experiences in technology supported learning 
environments: experiential learning, discourse or dialogic learning, supportive learning, learn by doing, critical 
thinking, conceptual change, and self-regulated learning. Each category contains a list of cognitive criteria and each 
criterion is operationally defined and cited. Results of using CATS to analyze a technology supported learning 
environment are described. Implications for using CATS as a design and evaluation tool are discussed. 

Introduction 

The purpose of this research was to develop and evaluate a tool called CATS or Cognitive Affordances of 
Technologies Scale that can aid instructional designers, e-learning specialists, and faculty in identifying and 
leveraging the teaching and learning affordances (instructional attributes) of a learning technology. Specifically, 
CATS can be used as a tool to support and promote the design of courses that enrich student learning by harnessing 
the pedagogical and cognitive affordances of a learning technology. We describe the methodology used to develop 
CATS and present the results of its application in a technology supported learning environment (TSLE). The TSLE 
was an upper level undergraduate Economics course that used the immersive virtual world Second Life and 
simulation software as learning technologies to augment classroom instruction. Educational implications and future 
research goals are also discussed. 

Theoretical Framework 

Several researchers and practitioners in higher education have argued that course design and delivery 
technologies such as Learning Management Systems (LMS) have consistently emphasized faculty dissemination and 
administrative tools over student learning tools and technological convenience over pedagogical effectiveness (e.g., 
Harasim, 1999; Marra & Jonassen, 2001; McLoughlin & Lee, 2010; Valjataga, Pata, & Tammets, 2011; van 
Harmelen, 2006). Additionally, researchers have found that most faculty use technology for administrative rather 
than instructional tasks because they don’t feel prepared or are not getting the technical or pedagogical support 
needed to develop effective instructional designs using technology (Dabbagh & Reo, 2011; Harrison, 2011). 
Compounding the problem is the fact that learning technologies are advancing at a rapid pace and understanding 
their cognitive and pedagogical affordances or instructional attributes is becoming increasingly challenging and 
complex (Milne, 2007). Furthermore, faculty, instructional designers, and e-learning specialists and administrators 
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are continuously under pressure from their respective institutions and organizations to deliver more courses and 
programs online and to integrate innovative learning technologies and digital resources into the teaching and 
learning process in order to address student and market demand (Dabbagh & Reo, 2011). Given these technological 
and pedagogical challenges the question we engaged was, how do we facilitate and accelerate the process of 
understanding the affordances of learning technologies in order to help faculty (and other stakeholders) design 
quality courses that enrich student learning experiences? 

The concept of affordances was first introduced by James J. Gibson in his 1977 article The Theory’ of 
Affordances. An affordance is a quality of an object, or an environment, that allows an individual to perform an 
action ( http://en.wikipedia.org/wiki/Affordance ). Greeno, Collins, & Resnick (1993) describe the inherent 
relationship between affordances and abilities as follows: “a situation can afford an activity for an agent who has 
appropriate abilities, and an agent can have an ability for an activity in a situation that has appropriate affordances 
(p. 1 14)”. This mutually exclusive relationship emphasizes perception and action rather than memory and retrieval 
and is considered an ecological approach to psychology or perceptual psychology (Hutchins, 2010; Greeno, 1994). 
According to Gibson, action and perception are linked through real-world objects that afford certain forms of action 
possibilities (affordances) for particular individuals or organisms (Albrechtsen, Andersen, Bodker, & Pejtersen, 
2001). Simply put, affordances are the interactions between users and tools. Gibson’s theory of affordances has 
direct implications on learning technologies because it emphasizes the non-neutrality of the learning space and 
prompts designers to consider the expectations and potentials that each learning medium brings forth to the teaching 
and learning process (Dabbagh, 2004). Affordance-based design has been used in a variety of disciplines however 
our focus in this research is on the cognitive affordances of a learning technology. A cognitive affordance is a design 
feature that helps, aids, supports, facilitates, or enables thinking and/or knowing about something (Hartson, 2003). 
Cognitive affordances are considered one of the most significant user-centered design features of learning 
technologies and have been shown to impact students’ learning (Graver, 1991; Wijekumar, Meyer, Wagoner, & 
Ferguson, 2006). 

For example, Allaire, Laferrier, & Gervais (2007) examined pre-service teachers’ perceptions or 
recognitions of the social and digital affordances of a networked learning environment designed to support and 
promote collaborative reflection and knowledge building activities. Social affordances were defined as human-to- 
human interactions mediated by technology such as an electronic forum and digital affordances were defined as 
human-machine interactions such as hard scaffolds (static learning supports that are embedded or built-in the 
software). The results revealed that social affordances were acknowledged or recognized by participants more 
quickly than digital affordances and that social affordances prompted more deliberative and critical levels of 
discourse than did digital affordances. Wijekumar et al. (2006) used two computer technologies to examine fifth and 
seventh grade students’ perceptions of their cognitive affordances; the first was an animated and game-like 
intelligent tutoring system designed using Flash to teach students a reading strategy and the second learning 
technology involved two types of chat rooms, agenda-driven and social chat. The results revealed that participants 
perceived computer technologies as communication tools, gaming tools, and as resources for completing homework. 
In other words, students in this sample population did not perceive computer technologies as learning tools. These 
results suggest that more research is needed to support affordance-based design that evokes meaningful and engaged 
learning. Hence, the focus of our research was to leverage the theory of affordances to develop a tool for identifying 
and purposively applying the cognitive affordances of learning technologies in course designs. 

Development of CATS 

CATS was developed in a doctoral course that examined the interaction between cognition and technology 
using multiple disciplinary perspectives including, cognitive science, psychology, neuroscience, education, design 
theory, instructional design, technology design, anthropology, sociology, information science, philosophy, 
semiotics, linguistics and other applicable fields. 

We used a highly inductive mode of inquiry while remaining focused on the core of the issue which is how 
to facilitate the understanding of cognitive affordances of learning technologies in order to support the design of 
quality courses that enrich student learning experiences. This qualitative inquiry process began by identifying 
organizing frameworks for cognition gleaned from the course readings (see Bransford et al., 2000; Glaser & Chi, 
1988; Greeno et al., 1996; Laurillard et al., 2000; O’Donnell et al., 2006; Rumelhart, 1980; Thagard, 1996; Sawyer, 
2006) which included behaviorist/empiricist; cognitive/rationalist; and situative/pragmatist. Next we identified 99 
cognitive criteria using these conceptual frameworks and organized these criteria using emergent themes through 
successive approximations and iterations. Initially, these themes included designing learning environments, 
constructing assessment, motivation, metacognition, self-regulated learning, and collaborative learning. 
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We then examined the cognitive affordances of the research-driven TSLEs provided in the course readings. 
Specifically, we examined the learning interactions enabled by these TSLEs and whether these interactions aligned 
with the themes and cognitive criteria we generated. This process yielded additional themes and cognitive criteria. 
We then revised and refined the themes and cognitive criteria resulting in the first draft of CATS which stands for 
Cognitive Affordances of Technologies Scale. A second draft of CATS followed after (a) revisiting the course 
readings for additional cognitive criteria, and (b) finding additional scholarly articles (e.g., Hartson, 2003; Jonassen 
& Land, 2000; Kim & Reeves, 2007; Graver, 1991) that address cognitive affordances of learning technologies. The 
current draft of CATS consists of seven categories and 41 cognitive criteria (see 

http://cehdclass.gmu.edu/ndabbagh/Resources/IDKB/CATS.html ). The seven categories are: experiential learning, 
discourse or dialogic learning, supportive learning, learn by doing, critical thinking, conceptual change, and self- 
regulated learning. Each category contains a list of cognitive criteria and each criterion is defined and cited as well 
as examples of learning technologies that can be used to support the criterion . For example, the category 
‘conceptual change’ involves instructor supported activities that purposely evoke a change in a student’s 
understanding of concepts and principles in the context of their existing knowledge; it is not a matter of simple skill 
acquisition or fact memorization (diSessa, 2006). The five cognitive criteria under the conceptual change category 
are: elicit prior knowledge, beliefs, and perceptions; bridge current idea to normative or new ideas; use pivotal cases; 
use anchoring experiences; and promote transfer. Each criterion is defined. For example, use of anchoring 
experiences provides a common experience from which a group can discuss and construct new knowledge (Krajcik 
& Blumenfeld, 2006). We used a wiki ( https://edit802falll0.pbworks.eom/w/page/28545057/EDIT-802) to 
document the iterative process of developing CATS. We also developed the following guidelines for using CATS as 
an analysis and design tool: 

(1) Begin by listing the design features of the learning technology you are using or thinking of using (e.g., 
an IMS); 

(2) Use CATS to observe a TSLE that uses this learning technology and analyze its cognitive affordances 
based on the following scale: 

• Used and observed: TSLE supports the cognitive ciffordance; it is used by the instructor and 
observable to the reviewer. 

• Used but not observed: TSLE supports the cognitive affordance; it is used by the instructor but it 
is not observable to the reviewer. 

• Not used but available: TSLE supports the cognitive affordance but the instructor did not use it. 

• Not available: TSLE does not support the cognitive affordance. 

(3) Identify the design feature(s) that engendered the affordance within the specific TSLE. 

CATS Applied to a TSLE 

Using these guidelines, CATS was applied to a TSLE to determine its viability in facilitating understanding 
of the cognitive affordances of learning technologies. The following subsections describe the selected TSLE for 
analysis, the learning technologies employed, the TSLE observation process, and the results. 

TSLE Description 

Economics of the Metaverse is an undergraduate, upper level, elective course requiring microeconomics 
and macroeconomics as prerequisite courses. The overarching course objective is to allow students to experience 
different intermediate level microeconomic concepts through role-playing so the student can ultimately discuss the 
concepts from a firsthand perspective. The course is intended to illustrate the applied and behavioral aspects of each 
concept using two learning technologies: simulation software and 3D avatar-based virtual worlds. The simulation 
software allows players (students) to participate in a variety of economic markets and games. For example, one 
game simulates a type of auction that allows players to try different strategies and experience bidding behavior 
whether one-on-one or one amongst a group. TerraEconomicus, a private Second Life island, provides a closed 
space for the students to access lecture slides, take quizzes, and socialize. Additionally, three private islands each 
support an economic experiment that allows the students as a group to experience the target concept. Besides these 
private islands, the course takes advantage of the public Second Life virtual world by directing students to specific 
locations to observe behaviors and actions associated with a specific economic concept. 

The course covers six economic concepts. The first five concepts are reviewed using lecture slides, a virtual 
world field trip, and a group discussion of the activities and assigned readings. Depending on the concept, different 
simulations and virtual world experiments were used to illustrate the concepts. In total, four simulations were 
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accomplished for each of four concepts and four virtual world experiments were conducted for three concepts. The 
sixth (last) concept only used a virtual world experiment and two class discussions. 

The class meets twice weekly in a computer lab for 1 .25 hours each session but in hybrid format. Although 
class sessions intended for lecture slides and field trips are designated in the course schedule, real-world physical 
class attendance is not required for these sessions. However, the room is still available with instructor present should 
the student prefer to use the computer lab to access the virtual world at that time. The remaining class sessions 
intended for the simulations, virtual world experiments, and class discussions require real-world attendance. 
Although the simulations and virtual world experiments are online, real-world attendance is required to ensure 
minimum technical difficulties. 

Students are randomly assigned to small groups (3 or 4 students) to visit the in-world lecture slides and go 
on joint field trips. Groups are re-assigned during the course. Individual written assignments include a 500-word 
paper per economic concept and a 10-page final paper. Group written assignments include providing questions for 
class discussion and a report for each field trip. 

Technology Description 

Two technologies were used in the course, albeit one having two instantiations to target two different 
purposes: the simulation software, the publicly accessible virtual world Second Life, and TerraEconomicus, a 
private virtual world using Second Life as the platform. The following describes each technology. 

Technology: Simulation Software. Multiple economic simulation games are available online as part of what 
is called the VeconLab (Holt, 2005) although the course uses its own version in order to retain data entries and 
results. The software supports customization by the number of participants such as for randomly selecting one-on- 
one bidding or allowing multiple players in a single game session. As shown in Figure 1, the software uses a simple, 
two-dimensional interface, in this case for a one-on-one auction game. Each game requires two or more players to 
submit ‘bids’ or ‘claims’ across a number of rounds that in turn impacts their ultimate payoff (win). The software 
records all data entries and winning results. 

Submit Decision for hound 5, ID: 8261 

Choose a claim between $0.80 and $2.00. Your earnings will be the minimum of your claim and that of the 
randomly selected person who is matched with you in this round, with an additional reward of $0.10 being 
paid to the person with the lower of the two claims, and a penalty of $0.10 being deducted from the 
payment to the person with the higher of the two claims. 


Round 

5 

Penalty 

Reward 

$0.10 

Your 

Decision 

Other's 

Decision 

Submit | 

Your 

Earnings 

* 

Cumulative 

Earnings 

* 

4 

$0.10 

$2.00 

$1.69 

$ 1.59 

$ 6.54 

3 

$0.10 

$2.00 

$1.75 

$ 1.65 

$ 4.95 

2 

$0.10 

$2.00 

$1.40 

$ 1.30 

$ 3.30 

1 

$0.10 

$2.00 

$2.00 

$ 2.00 

$ 2.00 


^ £con Lab - December 2. 2010 

Figure 1 VeconLab - Example Simulation Software interface 

Technology: Virtual World - Second Life. A virtual world is an online, persistent, 3D, interactive animated 
environments accessible by many users simultaneously (de Freitas, 2008; EDUCAUSE, 2006). Individuals, 
represented by avatars, control their in-world actions whether to move, communicate, collaborate, create, or 
socialize in-world (EDUCAUSE, 2006; Robbins & Butler, 2009). Many different kinds of virtual worlds exist, 
offering role-playing games, procedural training and simulation, and socialization. This economics course uses a 
social world, Second Life, which emphasizes communication and community building. Second Life requires login 
through an interface called a viewer. Second Life is a publicly accessible environment. In this course, students are 
directed to locations to observe economic behaviors. 

Technology: Virtual World - TerraEconomicus. TerraEconomicus is comprised of four islands created 
specifically for this course and for economic experiments. Although flying is supported on the main island, the 
experiments do not support flying as all movement is done by walking or running. The following summarizes the 
main island containing the skybox lectures and two experimental islands. The TerraEconomicus Main Island 
provides a covered auditorium-style seating area and areas to socialize such as participate in a game. Houses are 
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used to contain information and provide teleport from the house to the associated skybox lecture. Each economic 
concept has a house. The purpose of other parts of this island is not known as well as the objects found underwater. 
Figure 2 provides a view of one house (for orientation) and the covered auditorium-style seating area. 



Figure 2 TerraEconomicus Main Island - House for Orientation and Covered Auditorium- Style Seating 

Tiki Island is a separate area dedicated for experiments. For this island, the experiment premise is that each 
student is assigned a house on one of two sub-islands. Each student’s house is capable of producing two of four 
available colored seeds. Students can fertilize their two seeds differently to produce different amounts of each 
colored seed. Students are given a value for seeds, for example, two red seeds and one blue seed, may be worth a 
L$1.00 to them. However, they may be producing red seeds and green seeds. Hence, going to market to trade their 
seeds with other students is how the student earns money. It is designed so that the opposite island has more 
opportunity to produce a particularly valuable seed. Additionally, one student is awarded a key to the other island 
which then also becomes tradable. If the experiment runs repeatedly, the results should indicate that only producing 
one color seed and trade for the other color seed needed is economically advantageous, i.e., focus on niche markets. 

Hurricane Island is also dedicated to a particular experiment. In this experiment, each of the eleven students 
has a house on an island subject to damage causing hurricanes. Each student earns money at a rate commensurate 
with the amount of damage their house has; the lower the damage, the higher the earning rate. Weather stations exist 
that can provide 100% protection if manned properly, that is, manned with three or four students depending on the 
weather station; less students mean less percentage protection. Students can also choose to remain in their house to 
individually defend against a hurricane but at a reduced rate. Subsequent repair time reduces the time available to 
earn money. The students make the choice of uniting to defend their houses, perhaps taking turns to man a weather 
station, or individually defend their houses. 

TSLE Observation Process 

Six real-world class sessions were attended. These included observing the Second Life orientation class, 
one software simulation on auctions, two different Second Life experiments, and two class discussions. During the 
Second Life orientation class, the reviewer participated in-world for a brief period for a first person perspective. 

Four virtual world sessions were accomplished independent of the class, including completing the Second Life 
orientation class activities embedded in a skybox lecture, reviewing two additional skybox lectures on two different 
economic concepts (but not the quiz), and conducting a field trip to an instructor-provided virtual world auction site. 
Real-world class observation of the software simulation and the two virtual world experiments allowed the reviewer 
to peer over the shoulder of each participant to see what each person was doing from their first-person perspective, 
i.e. seeing what they were seeing. During all real-world class observations and independent virtual world sessions, 
the observer maintained notes on all activities; this included directions and comments made by the instructor as well 
as student comments and questions. Since this was a hybrid course that relied on real-world class time and two 
different technologies (simulations, virtual worlds), each technology was evaluated separately to determine whether 
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different cognitive affordances were supported. It was anticipated that the classroom may also offer different 
cognitive affordances, hence the classroom was evaluated as a separate learning environment based on class 
discussion as well as the affordances associated with the written assignments. 

Analysis of the TSLE Cognitive Affordances 

The cognitive affordances of the TSLE were evaluated using CATS. Evaluation meant noting if each 
cognitive affordance was: (1) used and observed, for example, feedback was provided automatically in the 
simulation and the virtual world while the instructor also provided feedback; (2) used but not observed, for example, 
collaboration was used but not observed, in that, the virtual world group field trips were accomplished outside the 
class that required a group paper to report findings but the activity was not directly observed; (3) not used but 
available, for example, independent exploration of Second Life could have been promoted but was not; or (4) not 
available, for example, the 2D simulations did not support collaboration or reflection. The results of this evaluation 
for each learning environment (class (Cl), virtual worlds (VW), and simulation (Sim)) are provided in Table 1. The 
analysis of the TSLE results for cognitive affordances is discussed by the seven major CATS categories. 

Experiential Learning 

In experiential learning, students are provided an authentic problem to generate hypotheses, gather 
information, and provide solutions, action plans, recommendations, and interpretations of situations (Dabbagh & 
Bannan-Ritland, 2005). This course used experiential learning extensively. The virtual world experiments afforded 
opportunities to problem solve. The experiments were scenario-based challenging the students to maximize earnings 
for the given conditions which represented an authentic but simplified economic context. Both the simulation and 
virtual world experiments supported hypothesis generation in that students developed action plans in accordance 
with their hypothesis on how to improve earnings. The technologies provided repeated cycles for students to act and 
react to test their hypotheses and re-strategize as appropriate. Additionally, the simulation and virtual world 
experiments provided opportunities to role-play as an auction bidder, a small business owner, and a home owner. If 
one considers the student’s development of new strategies as the generation of new ideas, then both the simulation 
and experiments supported this affordance. Similarly, the simulation and virtual world experiments supported 
experimentation through the ability to test new strategies. 


Table 1 - CATS Economics of a Metaverse Observation Results 


COGNITIVE AFFORDANCE 

Used, 

Observed 

Used, 

Not Observed 

Not Used, 

but Available 

Not 

Available 

Sim 

VW 

Cl 

Sim 

VW 

Cl 

Sim 

VW 

Cl 

Sim 

VW 

Cl 

Experiential Learning 













Problem Solving 


X 







X 

X 



Hypothesis Generating 

X 

X 







X 




Exploration 








X 

X 

X 



Role Playing 

X 

X 







X 




Generate New Ideas 

X 

X 







X 




Experimentation 

X 

X 







X 




Teacher-Guided Discovery 








X 

X 

X 



Inquiry-Based 








X 

X 

X 



Discourse/Dialogue 













Collaboration 





X 




X 

X 



Reflection 



X 





X 


X 



Multiple Perspectives 








X 

X 

X 



Articulation 



X 


X 





X 



Supportive (instructor or system) 












Coaching 








X 

X 

X 



Scaffolding 



X 





X 


X 



Modeling 


X 







X 

X 



Explaining 


X 

X 







X 



Feedback 

X 

X 

X 
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Task Breakdown 










X 

X 

X 

Use of Imagery or Visuals 


X 

X 







X 



Learn by Doing 













Personally Relevant 

X 

X 







X 




Authentic 

X 

X 







X 




Context / Situated 


X 







X 

X 



Build Artifacts 








X 

X 

X 



Critical Thinking 













Decision Making 

X 

X 







X 




Analysis 

X 

X 

X 










Synthesis 



X 







X 

X 


Critique 



X 







X 

X 


Construct an Argument 



X 







X 

X 


Conceptual Change 













Elicit Prior Knowledge, 
Beliefs, and Perceptions 






X 




X 

X 


Bridge Current Idea to 
Normative or New Ideas 



X 







X 

X 


Use Pivotal Cases 


X 







X 

X 



Use Anchoring Experiences 

X 

X 







X 




Transfer 



X 







X 

X 


Self-Regulated Learning 













Promote Metacognition 








X 

X 

X 



Goal Setting 








X 

X 

X 



Task Strategies 








X 

X 

X 



Time Management 



X 





X 


X 



Motivation - Intrinsic 



X 

X 

X 








Motivation - Extrinsic 

X 

X 

X 










Self-Monitoring 








X 

X 

X 



Self-Evaluation 








X 

X 

X 



Tally 

11 

17 

15 

1 

3 

1 

0 

14 

24 

29 

7 

1 


Although students strategized and observed the effects of their actions in an authentic environment, they 
did not frame questions to be solved. Therefore, exploration was not considered available in the simulation and 
experiments. However, Second Life might afford the opportunity for exploration. Teacher guided discovery and 
inquiry-based learning was also not available in the simulation and the virtual world experiments as it is not 
appropriate for the purpose of those specific instructional activities. Second Life could afford the instructor 
providing insights and guidance if so desired as well as possibly creating an inquiry-based problem to be addressed 
but that would need to be investigated. 

Discourse / Dialogic Learning 

Dialogic learning emphasizes social interaction through discourse, dialogue, conversation, and social 
negotiation (Dabbagh & Bannan-Ritland, 2005). The simulation did not support discourse or dialogic learning. The 
virtual world experiments did not support collaboration, articulation, or reflection. Although the virtual world 
experiments relied on students working cooperatively using text chat, it was to maximize earnings, not to achieve 
shared understanding. Collaboration, articulation, and reflection was however supported in the field trips since 
students were required to visit instructor-provided Second Life sites as a group to observe an economic behavior and 
subsequently provide a group report in response to instructor-provided questions. Since these field trips are 
conducted in-world at a time selected by the student group, it was not observed and unknown if the students 
discussed the questions while in Second Life or used other means. 

Reflection was afforded through the group field trip reports and the 500-word papers per concept as some 
instructor-provided topics were reflective in nature. For example, one question available for the paper inquired about 
the institutional features that were present or absent during the Hurricane Island experiment that promoted / 
inhibited coordination on group weather defense. Real-world class discussions were also reflective in nature at 
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times, requiring students to articulate their personal perspective. Multiple perspectives were also provided in the 
readings. Since a web browser can be opened in Second Life, other means to directly add reflection to the 
technology is possible such as accessing a blog or uploading documents. Multiple perspectives could also be 
afforded by providing multiple Second Life sites that exhibit different perspectives to an economic concept. 

Supportive Learning (instructor or system initiated/driven) 

These cognitive affordances are initiated by the expert, coach, mentor, instructor, or embedded 
performance support system, with the goal of modeling the desired performance, skills, or process, and observing 
and supporting learners during their execution of a learning task (Kitsantas & Dabbagh, 2010). The simulation only 
provided one supportive learning affordance, feedback. At the end of each simulation game, the student is informed 
of their earnings and also the highest earnings recorded. The student can mentally assess how well their strategy 
worked through comparison of results. This type of feedback does not address corrective action or suggest future 
strategies, indicating the feedback could be improved for deep learning. The virtual world experiments supported 
modeling and feedback. The experiments themselves were models of economic principles that allowed the students 
to experience those concepts. Similar to the simulation, feedback informed the student of their current earnings but 
without recommended improvement or alternative strategies. The TerraEconomicus skybox lectures provided 
explaining and the use of visuals through text and graphics. The graphics represented mathematical equations and 
relationships governing different economic concepts. During one class discussion, the instructor provided a graphic 
display of student results from the Tiki Island experiment to illustrate the results of their negotiating for the buying 
and selling their product (seeds) as compared to the theoretically predicted relationship. This same graphical 
comparative approach was used to illustrate the effects of personal valuation of goods relative to the community’s 
utility value. The instructor also used scaffolding by providing verbal hints to help the students articulate their 
thoughts and bridge to normative explanations. Task breakdown was not considered applied although the course 
syllabus breakout into six economic concepts might be considered as such. 

Learn by Doing 

In learn by doing, students apply the objective concepts and skills in a realistic activity which develops the 
student’s expert knowledge construction via experience (Cobb & McClain, 2006; Greeno et al., 1996; Koedinger & 
Corbett, 2006). The simulations, experiments, and field trips were relevant to the course but may or may not be 
personally relevant to the individual. With monetary generating web sites such as ebay and craigslist, and the notion 
of real-life negotiating for products such as automobiles, it would seem some activities would be considered 
personally relevant. However, for at least two students during the Hurricane Island experiment, the notion of public 
defense for home ownership did not seem personally relevant based on their observed behavior. One student chose 
to not earn money and expressing that the rewards did not warrant an effort to earn money. Another in that same 
experiment purposefully did not participate in the group defense fully recognizing this as aberrant behavior as 
indicated by comments made at the end of the experiment. That being said and assuming engaged participation 
indicates personal relevance then the activities were relevant to most students. It is unknown if attending an auction 
for horses (field trip) would be relevant to the students. In the end, it was decided that the simulations and virtual 
world activities were relevant as well as authentic, i.e., these types of economic decisions exist in the real-world. 

With regards to the cognitive affordance of context situated, the simulation was not set in a meaningful 
context; it was a simple computer game to illustrate auction activity. The virtual world experiments and field trips 
were context situated albeit the experiments were provided in a simplified context to highlight specific economic 
concepts. Real-life may embody these economic concepts but not in such a simplified manner by reducing 
influential variables. For example, during Hurricane Island, some students spoke about wanting a means to trade and 
negotiate for time spent in the group defense weather station. Others were noted as being opportunistic by reaping 
the benefits of group defense but not personally contributing time. In real-life, economic policy as well as personal 
interaction may promote entire group participation. These types of accountability balances and checks were not 
supported in the experiment. No activities provided an opportunity to explicitly externalize personal knowledge by 
building artifacts. 

Critical Thinking 

Critical thinking involves processing of collected information including behaviors reasoning, deciding, 
analyzing, synthesizing, critiquing, and arguing (Paul, 1995). Decision making and analysis were observed in the 
simulations and virtual world experiments as evidenced by the choices students made based on the information 
provided to them. The technologies themselves did not support synthesis, critique, or constructing an argument. 
However, the class discussion did provide the forum to analyze their behaviors within the context of the readings 
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and to synthesize into a more coherent representation of the economic concept. Additionally, the assigned papers 
provided another opportunity to develop critical thinking skills through analysis, synthesis, critique, and 
argumentation in written form. 

Conceptual Change 

Conceptual change involves teacher supported activities that purposely invoke a change in a student’s 
understanding of concepts and principles in the context of their existing concepts; it is not a matter of simple skill 
acquisition or fact memorization (diSessa, 2006). The instructor conducted a pre-course survey of student 
technological familiarity but it is unknown if student prior knowledge of the target economic concepts were 
solicited. The course requires a microeconomics and macroeconomics course as prerequisites but it is not validated 
as noted by the instructor. Additionally, eliciting for prior knowledge via survey may only reveal a student’s 
perception of their knowledge. Eliciting prior knowledge through student externalization is more indicative of their 
understanding (Bransford et ah, 2000). It is unknown if the first class included explicit exhibition of prior 
knowledge. The class structure and format of the simulation and virtual world experiments appears to preclude 
customization to student’s current knowledge (although the domain expert may best assess if customization is 
possible). The skybox lectures contained pivotal cases to support an economic concept through text and graphic 
representation. The course used multiple anchoring experiences from the simulation, virtual world experiments, and 
field trips that were made reference to during class discussion. Transfer was not afforded by the technologies, that is, 
there was no opportunity to re-apply what was learned to new contexts or situations. The concept on auctions could 
be supported if actual bidding were conducted during the field trip, that is, re-apply what they learned from the 
auction simulation. Through class discussion, the instructor promoted bridging the students’ articulated ideas to 
normative ideas, translating novice representations to expert terminology and at times using examples to transfer 
concept application across different domains such as buying an automobile as another example of price negotiation. 

Self-Regulated Learning 

Self-regulated learning is goal oriented actions that an individual uses to acquire knowledge and skills 
without relying on others; the learner orchestrates one’s own learning by planning, monitoring and correcting errors 
(Kitsantas & Dabbagh, 2010). The simulation and virtual world experiments only supported motivation, intrinsic 
and extrinsic, as affordances for self-regulated learning. Both technologies provided extrinsic motivation in the 
form of awarding monetary earnings per student. The earnings are paid to the student in Second Life Linden dollars. 
Second Life has a real economy with purchasable virtual items. Linden dollars can also be exchanged for US 
dollars; L$270 is worth approximately US$1 as per the instructor. Awarding student earnings was done purposely as 
motivation for students to actively participate in the simulations and experiments as opposed to passive 
participation. However, as one student noted, their current earnings was worth about US$0.30. While low in real-life 
purchasing power, it may be of more value in Linden dollars. So what might have the appearance of extrinsic 
motivation is actually dependent on a student’s personal valuation of the earnings. With regards to intrinsic 
motivation, the simulation and experiments may provide a challenge or curiosity that increases a student’s 
motivation through curiosity to investigate different strategies and determine their impact. Interestingly, the 
instructor verbally noted that students should be intrinsically motivated to actively participate in the simulation and 
experiments as they are a means to learning about economic principles. Lastly, the instructor supported time 
management by verbally stating upcoming assignment due dates during class time and also stated emails were sent 
class-wide with the same information. 

TSLE Cognitive Affordances - Conclusions and Recommendations 

Economics of a Metaverse provided multiple opportunities for cognition. In accordance with CATS, 29 of 
41 cognitive affordances were observed within the learning environment (albeit two not directly observed). Three of 
the seven categories were completely addressed by all cognitive criteria indicators. Three categories had at least 
61% of the indicators addressed. One category, self-regulated learning, was the least addressed with only three out 
of eight indicators addressed. Some of the 12 missed (unaccounted) opportunities for cognitive affordances may be a 
reflection of course content and learning objectives. For example, as the course content was not procedural or task 
oriented, the need for coaching may be limited. Additionally, if one’s epistemological belief holds that building 
artifacts is strongly related to deep learning, then the category of discourse / dialogic could be improved. 

It is important to note the cognitive contribution of each technology and the classroom to the overall 
cognitive affordances of the learning environment should the course be updated. The simulation contributed the least 
amount of cognitive affordance, with discourse / dialogic being completely void; indicating the need for these types 
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of simulations to not be left stand-alone activities without additional supporting cognitive affordances. The 
classroom (discussion and papers) and virtual world (experiments, field trips, and skybox lectures) provided nearly 
the same amount of cognitive affordances but were strong in different categories. The virtual worlds provided ample 
cognitive opportunities through experiential learning, supportive, and learn by doing. Where the classroom 
environment was void of experiential learning and learn by doing, it was however rich in supportive and 
discourse/dialogic opportunities. One could argue that the virtual world activities were classroom activities, but for 
the purposes of understanding the contribution of the individual technology components to the overall learning 
environment they were separated. 

Some simple changes could be made to this course to improve its cognitive affordances in accordance with 
CATS. Additionally, given that students (1) expressed concerns over lack of time to forge business partnerships and 
develop individual responsibility for group defense; (2) indicated Linden dollars was not a strong motivator; and (3) 
showed discontent with the lack of individual accountability in participating in group public defense, the course 
could be improved to address these student-perceived shortcomings. The following summarizes these 
recommendations. 

Discourse / Dialogic, Reflection: Although reflection is currently provided through group papers and 
possibly through individual papers, it could be more prominent as a requirement. For example, students could 
compare their selected strategies for improved earnings to that provided in the readings and why the results were 
similar or different. In this manner, the reflection affords the opportunity for the student to articulate their 
understanding of the concepts relative to their experience. Blogs had been tried previously but were found 
ineffective. It is unknown if there was poor participation or if the quality was subpar. It is also unknown if the blog 
structure and rubric posed barriers to successful implementation. 

Learn by Doing, Build Artifacts: Building artifacts was not afforded in this course although the possibility 
exists with some programming within TerraEconomicus. For example, the auction simulation can be run multiple 
times allowing each student to use a different strategy each game. Students can record their strategy and 
corresponding earnings in sequence and subsequently posted to TerraEconomicus for public review. Others can 
view the different types of strategies, the order of the strategies, and the winning strategies; all serving to provide 
multiple perspectives, motivation to be the high earner, and a means to externalize their understanding. 

Conceptual Change, Transfer: Classroom discussion was the only means of transfer. Although the concept 
on auctions might have supported transfer if actual bidding were conducted during the Second Life field trip, that is, 
re-apply the strategies they learned from the auction simulation. Alternatively, create a TerraEconomicus auction to 
accomplish the same opportunity for transfer but in a more controlled environment, that is, not participate in a public 
site where the items for auction are not controlled. Auction items might include a half-point on their final grade or 
100% protection card for during a hurricane. 

Self-Regulated Learning, Extrinsic Motivation: In addition to Linden dollar earnings, it may also increase 
participation to track and publicly record the highest earnings per course offering. One could create a ‘wall of fame’ 
within TerraEconomicus that includes the earnings, a photo of the avatar, and the avatar name. 

Self-Regulated Learning, Multiple Indicators: The class syllabus and a more explicit calendar could be 
posted in TerraEconomicus for persistent public viewing. The calendar can include class date, activities for that 
session, in-world/real-world location, and assignments due that session. Since participation is in part measured 
through class attendance, this information could be publicly posted using avatar names for anonymity (albeit 
anonymity may be questionable in a small class environment). 

CATS as a Design and Evaluation Tool 

In the Economics hybrid course, the 2D online simulations were found to support the least number of 
cognitive affordances based on CATS with the discourse/dialogic category being markedly absent suggesting that 
other learning technologies are needed to evoke additional cognitive affordances. The classroom and virtual world 
experiences provided nearly the same degree of cognitive affordances but in different categories of CATS. 
Specifically, Second Life provided ample experiential learning and learn by doing while the classroom experience 
was rich in supporting discourse/dialogic learning. 

CATS allows the instructor or instructional designer to examine the cognitive affordances of a TSLE by 
observing and analyzing the extent to which each cognitive affordance is being engendered or invoked. The goal is 
to improve the cognitive design of a TSLE in order to support purposeful and meaningful learner actions and 
learning interactions. By better understanding the cognitive affordances of a learning technology, instructors and 
instructional designers can produce more effective student centered instructional designs. 
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This exploratory research revealed that CATS has the potential to become a highly effective tool for 
understanding the cognitive affordances of a learning technology and assisting faculty in purposively and 
intentionally taking advantage of these affordances to develop effective, engaging, and enriching TSLE. This has 
implications to all faculty, instructional designers, and e-learning developers and administrators involved in using 
learning technologies to support the design and delivery of instruction. Specifically, this research revealed that 
CATS can be used to: 

1. Evaluate/analyze the cognitive affordances of a technology supported learning environment 

2. Purposively increase the cognitive affordances of a TSLE 

3. Address cognitive affordances as a dimension of instructional design 

4. Design instructional strategies and learning activities that align with the cognitive affordances of a 
particular learning technology 

5. Train faculty and instructional designers to effectively integrate technology into the teaching and 
learning process and enrich student learning experiences 

Further research is needed to evaluate the validity and reliability of CATS as a design and evaluation 
instrument. Specifically, content and construct validity will be examined by re-examining the literature on 
affordance -based design and applying CATS to a range of learning technologies in a variety of educational settings 
and using interrater reliability for each instance. 
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Training for Library Site Supervisors 
The Significance of Mentorship 

Kaye Dotson 

Abstract 

This study examines an integral component of the online facilitated internship. Online education is 
increasingly prevalent in graduate schools of teacher education. The program under review by the researchers is 
completely online and is based on goals and objectives established by the Department of Library Science (DLS) 
faculty through the curriculum development process. This study was initiated to gather data to improve the 
program’s clinical experience, the professional internship that is facilitated online. Technology played a strong role 
in this study, which focuses upon the need for support of site supervisors by university programs, as the program 
under review is offered entirely online. 


Introduction 

Distance learning is rapidly becoming the mode of choice for delivering education and instruction. The 
physical presence of the teacher or professor may not be possible, therefore it is important to ensure quality of the 
overall course. Specific components of programs require close examination. The clinical experience, particularly, as 
the crucial link between theory and practice, is perhaps the most difficult program component to supervise and 
facilitate in an online environment and the need for examination of aspects of the online facilitated internship is 
clear. The internship, as the clinical experience, offers a culminating opportunity to put into practice the theory 
learned in coursework, as well as an avenue to experience professional work while continuing to have access to 
academic support. Funding may not be available to support faculty travel for field placements, but still an 
environment conducive to training, modeling, and nurturing the pre-service librarian is required. It is critical that the 
graduate school internship unambiguously prepare future professionals for the impact of evolving professional 
responsibilities and the impact of the on-site supervisor is critical. Particular attention and effort are required to 
maintain contact and support with and for site supervisors who are working in distant schools and sites. 

The goal of this study was to understand the perspectives of supervisors who have experienced an online 
supported internship. Understanding the perceptions of site supervisors as integral participants in the internships, 
and making advantageous use of the technology available today is essential for continuing program improvement. 

An online survey was designed and distributed to site supervisors who had previously served a library science 
program in an online facilitated internship for the purpose of gathering perspectives of librarian site supervisors. 

Context 

The program under review is part of the College of Education at East Carolina University and is one of the 
38 AASL/NCATE endorsed school library programs throughout the US. The Library Science program, approved by 
North Carolina’s Department of Public Instruction, is completely online and is based on goals and objectives 
established by the Department of Library & Information Science (DLIS) faculty through the curriculum 
development process. This process evolved through systematic planning by the DLIS Curriculum Committee in 
response to requirements for licensure from the state Department of Public Instruction (DPI) and the Standards for 
Initial Preparation of School Librarians designed by ALA/ American Association of School Librarians (AASL). 
Based on the program goals and objectives, the curriculum addresses the theory, principles, values and practices 
necessary for MLS students to succeed in their professional library careers. The MLS program fosters curricular 
goals and objectives through a variety of educational experiences, culminating in the capstone clinical experience, 
the professional internship. The internship is a crucial, culminating link between theory and practice and depends 
heavily upon the expertise of on-site supervisors. 

Research Questions 

The main purpose of this study was to examine the perceptions of site supervisors regarding the roles and 

responsibilities of those serving interns in the field. The research questions were formulated as: 
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1. What were site supervisors’ perceptions regarding predisposing, enabling, and reinforcing 
domains for the online facilitated internship program? 

2. Were there significant relationships between the three domains and the success of the internship 
program? 

3. What did qualitative data clarify for us regarding the site experience? 

Method 

A mixed methods design served to collect both quantitative and qualitative data. This study used a 19-item 
web-based survey, “Internship Site Supervisor Perceptions Survey” (ISSPS), to gather data. ISSPS included both 
open- and closed-ended questions. The ISSPS included 15 closed-ended questions and four open-ended questions in 
order to collect data on site supervisors’ perceptions regarding the online facilitated internship program. Four 
questions asked participants to report their leadership position, working setting, certified librarian status, and 
supervisor status. Ten questions tapping predisposing domain, enabling domain, and reinforcing domain were 
measured with a five-point Likert scale (1 = Strongly disagree, 5 = Strongly agree), excepting one question which 
asked the length of the internship and used a dichotomous response option.. Data were collected from a pre- 
determined population including certified librarians nationally, but predominantly from the southeastern United 
States, as this was the area in which most of the interns served. Individuals who have served as site supervisors for 
online facilitated internships were invited to participate in the confidential survey. The research population was 
accessed through the cooperation of fieldwork supervisors affiliated with the graduate school under review. Site 
supervisors included individuals from public schools, including large, small, rural, and urban. Surveys were sent to 
site supervisors at the end of each student internship experience. Responses to the survey were examined through the 
lenses of situated cognition theory and the Precede-Proceed model. 

Table 1: Items used to assess leadership position, working setting, certified 
librarian status, and supervisor status of site supervisors 


Items 


1 . Did you serve as a site supervisor? 

2. Are you a certified school librarian? 

3. In what setting did you work? 

4. What leadership positions, within your setting, do you hold? Please list 
them below. For example, are you serving as a team leader, project leader, 
etc. 

Table 2: Items used to assess predisposing domain, enabling domain, and 
reinforcing domain 


Items 


5. Do you feel the length of the internship was adequate as a capstone 
experience for preparation for librarianship? 

6. The expectations of an internship supervisor were made clear by East 
Carolina University’s Department of Library Science. 

7. I knew who to contact if I had questions about my intern. 

8. I received appropriate and timely responses from East Carolina University’s 
Department of Library Science regarding any questions I had. 
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9. The internship process went smoothly in my opinion. 

10. Both the intern and my school/organization gained something of value from 
this experience. 

11. Based on your experience with our student(s), would you take another 
intern from the Department of Library Science at East Carolina University? 

12. My role in assessment of my intern's performance was made clear to me. 

13. Please list any specific areas you feel our graduate program should focus 
upon to prepare you as a site supervisor. 

14. Please list any specific challenges to you in your service as a site supervisor. 

15. I had necessary skills to mentor my intern in best practices, including 
appropriate uses of resources and technologies. 

16. I felt comfortable with this experience 

17. I feel additional preparation for site supervision would have helped me with 
this experience 

Table 3: Items used to assess open-ended questions on specific areas graduate 
internship program should focus for site supervisors’ preparation, necessary 
skills and challenges of being a site supervisor, and any comments about 
experience with the intern. 


Items 


18. Please list any specific skills in the order of importance that you believe our 
graduate planning program should focus on to prepare our students for 
professional positions 

19. Please note any other comments about your experience with the intern that 
you would like to share: 


Theory 

Situated Cognition theory is based upon the perception that learning is constituted through the sharing of 
purposeful activity (Brown, Collins, & Duguid, 1989) in the preparation and practice of future school librarians. The 
teaching and learning of library science skills in real-life context offers the opportunity for graduate schools to do a 
more effective job in the preparation of future professional school librarians (Dequoy & Stefl-Mabry, 2012). 

Situated cognition offers a theoretical foundation for examination of the role of the site supervisor for teaching and 
learning in the real world context. 

This study was further examined through the lens of the Precede-Proceed model. The model has two main 
components, the PRECEDE and the PROCEED (Gielen, McDonald, Gary, & Bone, 2008). This is the acronym for 
“Predisposing, Reinforcing, and Enabling Constructs in Educational F, n v iron m e n t a I Diagnosis and Evaluation.” 
Those constructs are factors that influence the behavior change process. Predisposing factors include individuals’ 
knowledge, skills, beliefs, etc. Reinforcing factors are those rewards or incentives that motivate people to continue 
existing behaviors, such as vicarious reinforcement, peer influence, etc. Enabling factors include programs, services, 
policies, resources, etc. Site supervisors’ perceptions of their roles, responsibilities, and online facilitated internship 
programs can be classified into three domains: predisposing, reinforcing, or enabling. As these domains have direct 
and indirect effect on the success of internship programs they were under review in this study. 
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Findings 


Of the survey participants 42% served in primary schools, 24% served in intermediate schools, while 32% 
served in secondary school settings. Leadership positions held by participants included principal, committee chair, 
manager, team leader, director, school improvement team member, etc. 

The length of the internship was satisfactory to 89% of participants. In regard to the clarity of expectations 
84% agreed that expectations were made clear to them by the university supervisor. Site supervisors reported only 
59% reported receiving appropriate and timely responses to questions. However, it may be noted, that 40% indicated 
neither agreement nor disagreement in this area. 92% indicated that the internship, overall, proceeded smoothly. 

Site supervisors reported, overwhelmingly, that they had gained something from the experience with 93% reporting 
agreement or strong agreement in response to that question 

The analyses of open-ended responses yielded six themes grouped in predisposing, enabling, and 
reinforcing domains: technology skills, collaboration/communication skills, direct contact, adequate information, 
time, and positive feedback. When asked about any specific area the graduate program should focus upon to better 
prepare site supervisors, respondents focused on two themes: direct contact from the university with the site 
supervisors sharing adequate information about roles, expectations, internship design; and expectations and 
guidance regarding the time commitment of interns. For example, one stated, “I would recommend direct contact 
with the site supervisor as to role and expectations of the program.” The response also implied that additional 
information was needed by the site supervisor. As another participant explained, “information prior to the beginning 
of the internship would have been helpful: what to expect, how many hours, what types of items was the intern 
expected to perform within the internship time period.” 

Conclusions 

The results of this study pointed to responsive program improvement for graduate educators to enhance 
support of on-site supervisors and interns in order to strengthen the online facilitated internship and to impact 
positively upon the profession. In general, the current study points to the need for development of a site supervisor 
training program to help prepare future site supervisors for the expectations of the role and to enhance 
communication between university program and internship site supervisor. To professionalize and support the on- 
site supervisor more effectively, ongoing professional networking opportunities are strongly recommended. These 
experiences will help with role preparation, evaluation of the intern, and understanding of the program. A 
mandatory prerequisite for site supervisor training would assist in the development of a sustainable pool of well- 
prepared, skilled site supervisors. The significance of this essential role should not be overlooked. It is 
recommended that site supervisor training be developed, implemented and followed by longitudinal research aimed 
at following and observing site supervision for program improvement. Researchers for this study are currently 
responding to results of the study by developing a proposal for a program to provide training for site supervisors 
based upon data from this study. The findings of this study will lead to the updating of guidelines for on-site 
supervisors for the program under review. Future plans towards these goals will be further discussed in the 
presentation session. 
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Abstract 

This paper describes the distance learning research activities and findings of a team involved in the 4-VA initiative. 
This initiative, sponsored by the State of Virginia, is designed to foster meaningful collaborations among four 
institutions with potential to “work together and, as a result, achieve more” (http://www.4-va.org). The researchers 
forged new ground by initiating a study of STEM learning spaces and inter-institutional collaboration. Findings led 
to meaningful and practical recommendations. 


Background 

This paper describes the distance learning research activities and findings of a team involved in the 4-VA 
initiative. The initiative, sponsored by the State of Virginia, is designed to foster meaningful collaborations among 
four institutions with potential to “work together and, as a result, achieve more” (http://www.4-va.org). Participating 
institutions are George Mason University, James Madison University (JMU), University of Virginia and Virginia 
Polytechnic Institute and State University (Virginia Tech). The mission is to make instruction more accessible and 
cost effective; to foster research competitiveness; and to develop courses and programs that lead to student success 
in Science, Technology, Engineering and Mathematics. 4-VA has many parts including but not limited to 
collaborative course design and development, and collaborative research among faculty at participating universities. 

Distance learning is a key area of concern for those involved in the 4-VA initiative since inter-institutional 
collaboration requires effective remote connections and interoperability. Each of the four participating institutions 
uses Cisco TelePresence systems that are connected to the National LamdaRail. The initiative has evolved beyond 
the constraints of a standard TelePresence room, though. To support meaningful distance collaborations in a more 
distributed space for STEM students and faculty is the core context of this study. 

Problem 

At participating institutions, faculty in Science, Technology, Engineering and Mathematics (STEM) were 
invited to use state-of-the-art TelePresence spaces for inter- institutional course delivery. In these rooms, instructors 
and students appear on large screens in the front while presentations are shown on adjacent monitors. The virtual 
shared space is designed to simulate the same physical space; for example, monitors are wide and positioned at table 
level, cameras are microphone activated, and lighting is consistent at each site. Unfortunately the room 
configuration, where seating and microphones are fixed, limits teaching and learning activities in student-centered 
labs in STEM content areas. In these areas, “problem-based learning (PBL) offers a viable method for teaching 


417 


inquiries” (Mong & Ertmer, 2013, p.16). Instructors facilitate the learning process and scaffold experiences to foster 
inquiry and discovery among students in authentic contexts. 

Faculty at one of the universities identified an alternate lab space for STEM course delivery. However, the 
room was unfurnished and without tools to foster best practices. There was no intentional design for STEM learning. 
The design challenges involved the optimal utilization of physical space as well as the virtual space, and the 
compatibility of technologies, strategies and design decisions employed by the collaborative institutions. 

Purpose 

The purposes of this applied research study included: ( 1 ) understand the theoretical and pedagogical nature 
of teaching and learning in the postsecondary STEM classroom and laboratory; (2) investigate the design of learning 
spaces that support innovative practices for STEM collaborations at distance; and (3) inform meaningful 
recommendations for equipment purchase and room redesign as part of the 4-VA grant initiative involving 
collaborative learning spaces at participating institutions in the State of Virginia. 

According to Hannafin and Land (1997), "All learning environments, explicitly or tacitly, reflect 
underlying beliefs about how knowledge is acquired and used” (p. 172). The researchers interpreted results of this 
study to propose an intentional design for the learning environment while keeping in mind that “The character of 
[learning] space is defined by a total experience; it is the combination of physical design and behavioral norms — 
and, more recently, technology interfaces — that define place” (Milne, 2006, 11.8). 

Methods 

The research was intended to answer the following questions: (1) how is teaching and learning currently 
conducted in STEM lab spaces? and (2) how can an existing STEM learning space be redesigned to promote student- 
centered approaches, interactivity, and effective distant collaboration with another institution? After conducting a 
literature review in this area, the investigators designed an applied study approved by the Institutional Review Board. 
Following the protocol, in addition to administering a STEM faculty questionnaire, interviews and focus group sessions, 
the researchers observed and tested existing systems and technologies at the local site and at a partnering institution. 

The 4-VA Director identified twenty STEM faculty whom the researchers invited to participate in this 
study. Two participated in the online questionnaire. Data from the questionnaire were used to generate additional 
questions for the semi-structured interview and focus group protocols. Five participated in an interview or focus 
group session. The participants were faculty in Engineering, Physics and Computer Science departments. Because 
participants represented a variety of STEM disciplines and had extensive experience in the field and at the 
institution, researchers were able to gain meaningful insights into the STEM lab needs and best practices. 

To understand existing systems, technologies and models, the researchers visited several local sites in 
addition to a model classroom at the remote site. They observed courses in the existing TelePresence rooms and 
connected to the TelePresence server from various campus locations using Cisco Jabber on a variety of devices. 
Remote connection to the server could allow the STEM lab access to the LambdaRail network. They visited a 
Center for the Performing Arts that offered state-of-the-art audio and acoustics systems in addition to a technique for 
routing sound from one room to another, which could prove useful for connecting the available lab to a nearby 
classroom to extend the learning space. At a football stadium control room the researchers observed a 360-degree 
camera zoom from atop one scoreboard to a close-up view of a blade of turf grass at the far end of the field. The 
flexibility and power of the camera could be useful in a lab space where students would work in a distributed 
manner throughout the room, with small and large objects. To consider alternative projection methods in the lab, the 
researchers visited Science on a Sphere where projectors around the room cast an image on a sphere in the center of 
the room rather than on an outer wall. This, again, could offer useful viewing space in a distributed room. Finally, 
the researchers visited a Scale-UP (Student-centered Active Learning Environment with Upside-down Pedagogies) 
classroom at a partnering institution. The strength of this model is that students work collaboratively with one 
another and with the instructor in an active, student-centered space. 

The researchers were expected to report findings to administrators on campus. To do so, findings would 
need to be interpreted and translated into collaborative technologies, practical techniques and configurations for 
implementation. The team identified and tested on-site and at a distance with faculty at all 4-VA institutions, a wide 
range of potential software and hardware solutions including but not limited to Beam (suitabletechnologies.com) 
remote presence system, the Double (doublerobotics.com) mobile telepresence robot, Kubi (revolverobotics.com) 
robotic telepresence mounts, Zoom (zoom.us) cloud videoconferencing solutions, Panasonic Toughbooks and Apple 
iPads using various apps. 
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Analysis 


The researchers organized the qualitative data into 4 themes that emerged during data collection: (1) non- 
pedagogical needs, (2) pedagogical needs, (3) collaboration and (4) problems. Then, they reviewed notes line-by- 
line to be sure each line was addressed in the initial recommendations. They combined and reduced 
recommendations to avoid redundancies, marked redundancies as areas of emphasis, and continued this process until 
all notes were combined and reduced. They matched non-pedagogical, pedagogical and collaboration needs with the 
features and affordances of technologies they observed and tested with attention to technologies, configurations and 
techniques that might solve identified problems. 

Findings and Interpretations 

The researchers incorporated all findings and preliminary recommendations from the data collection 
process into a solution-oriented presentation that drew clear connections among the background, methods, outcomes 
and preliminary recommendations for STEM lab redesign at the local site. They matched technological solutions to 
identified needs, problems and potential collaborative activities; and presented preliminary recommendations to 
administrators and staff in classroom services, computing, networking, instructional technologies and the 4-VA 
initiative. Findings responded directly to the research questions. Key findings related to videoconferencing, lab 
space, projection and display, computing and mobile devices. 

V ideoconferencing 

Any videoconferencing solution should allow for two-way interactive audio and video as well as remote 
camera and screen sharing. It should be able to clearly capture small objects like computer board wiring, as well as 
large and mobile objects like workstations and robots on the floor and in the air. The solution would need to capture 
objects from multiple angles. Faculty and students would use the solution to connect to community members, public 
schools and other universities via individual and group videoconferencing strategies. They would need to share 
screens on mobile laptops and other devices, and display information using wireless projection from any location in 
the room. Remote instructor and students would need to feel connected as if they were together in the same 
classroom. After extensive testing, Beam was proposed as a solution for creating a sense of presence between 
remote instructor and students. With Beam the instructor can virtually navigate the room, look over students’ 
shoulders, and offer immediate feedback and input, engaging in real-time conversations with students. For station 
top display, the researchers recommended Kubi, a robotic mount that can tilt, pan and elevate any tablet. A remote 
or local instructor can control Kubi from a mobile device. Mounted on a workstation, one Kubi with mobile device 
and an application like Zoom.us can pan and tilt to display tabletop work and facilitate student remote 
videoconferencing from the workspace. One might use a mobile device to display small and large objects. iPads are 
suggested for high-quality visuals and collaboration while ToughPads are suggested for rugged use and some 
productivity. 

Lab Space 

The physical lab space will be shared among Science, Technology, and Engineering and Mathematics 
faculty. It measures 26’ wide x 32’ long, which should be spacious enough for workstations, large bikes, and small 
and large robots that maneuver the floor and air space. Furniture should be flexible and movable where tables can 
fold and be moved against the wall, or combined to create larger workspace areas. The lab space in this study should 
accommodate 20-25 students at most. Each student workstation would accommodate two students, requiring a 
power supply with four to five retractable outlets pulled from the ceiling to support the movement of flexible 
furniture in the room. If using monitors or mobile devices for display, adjustable mounts could be added to the 
workstations. 

Projection and Display 

In this case, an effective configuration for projection and display would enable instructor and student work 
to be broadcast without requiring a ‘front’ to the room. A ceiling projector that swivels could project on any wall. 
Technologies like LeapMotion would allow for swiping from mobile device to a shared display. The features of 
tools that support augmented reality like zSpace ( http://zspace.com/solutions/education/ ) and Google Glass 
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( http ://www. google. com/ glass/ start/ ) or a competitor would add perspective and mobility in the distributed lab 
space. 

Computing and Mobile Devices 

The use of mobile devices could solve very practical problems of navigating the physical space and sharing 
information in the virtual space. With students in control of the mobile device camera and other features, capturing 
visuals from various angles and in an authentic context is manageable and meaningful. Faculty participants wanted 
to explore discipline-specific software to help students move from concrete ideas to abstractions that may be 
difficult to grasp otherwise. They wanted to use Cloud computing for subjects involving GIS activities, for example, 
where students collect and report data from the field, and hoped to become more environmentally friendly by 
incorporating digital notebooks rather than requiring printed materials. 

Conclusion 

The purpose of this study was to address complex issues of inter-institutional collaboration in STEM 
content areas. Underlying the findings and recommendations were theoretical and pedagogical foundations for the 
design of student-centered learning environments. Because the lab work is distributed, flexible and connected to a 
remote site, key solutions related to the design of the lab space, its furniture and configuration, and to mobile 
solutions for videoconferencing, display and computing. While conceptual in nature, preliminary recommendations 
for the collaborative offer worthy solutions to identified needs and problems. Further research is needed after 
implementation, to understand the learning experience in this environment. 
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Abstract 

This paper captures the experience of an international team of novice instructional designers who were 
united through a service-learning cognitive apprenticeship. The team developed a series of online modules for a 
client that provides instructional resources to assist international non-profit organizations serving the development 
sector. The AECT’s call for submissions to the AECT IAP Distance Education Best Practice Award provided an 
opportunity for team members to reflect on their development as instructional design practitioners in the face of real- 
world challenges. Best practice strategies for employing a cognitive apprenticeship model and leveraging team 
diversity in order to navigate the ill-structured nature of real-world instructional design are provided. 

Introduction 

Pairing service-learning with a cognitive apprenticeship (CA) model of instruction can strengthen 
communities, enrich disciplinary learning (Bringle & Hatcher, 1996; Jundt, 2010), and breathe “new dignity” into 
scholarship (Boyer, 1994, p. A48). Service-learning, a pedagogical approach that integrates academic and 
community-based learning, provides an authentic context for student engagement (Jundt, 2010), “promotes theory- 
practice connections, enhances student motivation and retention, utilizes cross-disciplinary perspectives, and 
develops workplace skills” (Zlotkowski & Saltmarsh, 2004, p. 42). Similarly, through CA, students are enculturated 
“into authentic practices through activity and social interaction” (Brown, Collins, & Duguid, 1989, p. 37) in which 
the invisible thinking of experts becomes more visible to novices (Collins, Brown & Holum, 1991). 

An instructor led service-learning CA supports the development of novice instructional designers and 
encourages the “integration of the intellectual and the moral in [preparing students] for the many roles of the scholar 
- discovery and synthesis, teaching and service” (Shulman, 2008, p. x). As such, in addition to sharing the 
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pedagogical benefits and strategies of a CA model, the team seeks to encourage universities to refocus on the 
original mission of contributing to a broader community by supporting student scholarship through service-learning 
(Kezar, Chambers, & Buckhard, 2005). 

Serving Non-Profit Organizations in the Development Sector 

This service-learning project was completed at a distance for a client that provides instructional resources 
to assist over 80 international non-profit member organizations in the development and relief sector build staff 
capacity. Seeking “to improve the lives of the world’s poor and marginalized communities,” the Seattle based client 
partnered with graduate students enrolled in a Professional Instructional Design course in Virginia to design and 
develop e-learning modules to prepare learners for the Project Management in Development (PMD Pro) Level 1 
Certification examination (LINGOs, 2012). The instructional content included the following: 

• Module 1: Introduction to Project Management 

• Module 2: Project Identification and Design 

• Module 3: Project Set Up 

• Module 4: Project Planning 

• Module 5: Project Implementation 

• Module 6: Project Monitoring, Evaluation and Control 

• Module 7: End of Project Transition 

The students were directed to develop five of the seven modules, with the remaining modules (Module 3: 
Project Set Up and Module 7: End of Project Transition) to be developed later that year. Target learners for the self- 
directed e-learning training modules are diverse, and include males and females aged 20 to 55 years old who speak a 
range of arterial languages and may have multi-lingual abilities. They serve as volunteers or employees for non- 
profit organizations located in developed and developing countries and possess varying levels of project 
management skills. They have the ability to perform basic computer functions including getting on the Internet and 
using basic office productivity software and hardware. 

There were many challenges to consider in preparing to deliver instruction to the target learners. Many lack 
or have intermittent Internet access, speak indigenous languages or require that instruction be localized before 
release. To overcome these challenges, the client deploys instruction in multiple delivery modes including self-led 
e-learning, blended, and face-to-face. Content is made accessible at a designated online site, the member 
organization’s local server or on compact disc. Member organizations have access to instructional source fdes to 
support customization. 

Developing Instructional Design Practitioners Through a Cognitive Apprenticeship 

One of the hallmarks of this project was the instructor’s ability to lead the class through CA, a model of 
instruction which requires extensive planning and is best suited for complex, ill-defined tasks (Casey, 1996; Collins, 
Brown & Holum, 1991; Darabi, 2005; Driscoll, 2005; Farmer, Buckmaster, & LeGrand, 1992; Ghefaili, 2003; 
Herrington & Oliver, 2000). CA ties the learner and expert in a mutually dependent relationship and emphasizes a 
community of practice in which one becomes part of the process of negotiating meaning (Wenger, 2010). Therefore, 
the instructor is not expected to play the role of ‘expert’ all the time. This project was ideal for the CA model as the 
class activities evolved based on interactions with a real client and the instructor acted as facilitator. 

The CA model consists of four overarching components: content, sequencing, method, and sociology 
(Collins, Brown & Holum, 1991). Content addresses the types of content needed for expertise, ranging from domain 
specific concepts and procedures to metacognitive control strategies which direct decision making though processes. 
Sequencing relates to ordering instruction to support the greatest level of understanding and includes such strategies 
as addressing the general before the specific, gradually increasing the level of complexity and varying the types of 
application tasks. The most popular CA component is method which addresses techniques for expertise development 
such as modeling, coaching and scaffolding. Sociology addresses the social aspects of learning such as 
contextualized or situated learning which allow learners to set their own goals and create a sense of community 
through cooperation. Content, method, sequencing and sociology components all work together to create a 
successful CA program. 

Domain specific content was developed through teacher led and self-directed instruction regarding ID 
project management principles. In addition, as a prerequisite to the course students were required to take courses in 
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foundational ID principles and applied theories. Heuristics were developed during domain specific instruction where 
the instructor modeled through authentic examples and student collaboration. The project was sequenced from the 
general to the specific and students were immersed in domain specific content before meeting with the client. 

The instructor modeled instructional design best practices throughout the course. Examples are described in 
more detail in subsequent sections of the paper and included transitioning from instructor to student directed 
meetings and asking questions rather than simply providing answers. Both strategies allowed the students to develop 
skills while simultaneously building confidence. 

Intrinsic motivation, which falls under the sociology categoiy, was integrated very early in the course as 
learners were asked to complete a strengths and interest survey created by the instructor and responses were used to 
place students in diverse module groups as well as provide students with opportunities to develop their desired skill 
set. All team members developed instructional design storyboards and turned them into the instructor for individual 
feedback. The instructor led a team discussion so best practices could be identified and incorporated into all the 
modules. This process inspired team members to actively contribute. 

The sections to follow offer a brief account of a CA model put to practice. Specific strategies that the 
instructor employed to support the development of the team’s project management and instructional design skills are 
provided. 


Development of Project Management Skills 

Poorly structured communication guidelines can impact a virtual team’s success and instructors are advised 
to implement formal processes (Dittman, Hawkes, Deokar, & Sarnikar, 2010). During the first semester, a Matrix 
Project Team Structure was created with each team member assigned responsibility for a specific function within a 
specific module team (see Figure 1). The 15 team members are represented as circles in the matrix below. During 
class students spent time in both functional (Project Manager, Designer, Developer) and module (1, 2, 4, 5, 6) 
meetings. 



The first semester was spent developing content knowledge. Every week for the first two months, students 
read chapters from an instructional design project management textbook and reflected on lessons learned. 
Simultaneously, they applied the readings to develop project management documentation through the support of job 
aids and student brainstorming sessions. The first client deliverable was the Project Proposal which was organized 
by the Project Management Functional Team. Similarly, the Design Teams and Development Teams were busy 
reviewing course content, developing instructional strategies and learning new software. Additional documentation 
developed by the student teams included a Communication Plan, Deliverable Review Form, Meeting Agenda 
Template and Status Report Template. A MS SharePoint site was developed to share digital assets and archive 
working drafts. The Status Report served as a critical communication tool between each team member and the 
instructor. Reports were turned in on a weekly basis and included tasks, hours, challenges and goals attainment. The 
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reports supported the instructor in assessing the student’s development and team contribution and she met with team 
members privately to provide individualized guidance. 

In the beginning, the instructor required the team to hold weekly client videoconferences via Skype. The 
team also shared profile pictures and bios with the client to support working relationships. Initially, the instructor 
modeled client relations. After each meeting students were asked to reflect on the lessons learned, articulate their 
understanding, and brainstorm ways of moving forward. In this case the complexity of the task that students had to 
undertake increased. As the class progressed students were scaffolded to manage meetings on their own. 

Resources to support authentic learning are often a challenge for CA application (Collins, Brown, & 

Holum, 1991; Ghefaili, 2003). This was not experienced by the team because the client’s sponsors granted team 
members access to Articulate Studio software and eLearning Art digital assets free of charge. 

The team experienced some of the challenges of real-world instructional design as they became challenged 
in grasping the content, aligning over 100 learning objectives and associated assessments with modules while 
simultaneously developing instructional strategies. The instructor revised the course on multiple occasions during 
the semester as the deliverable schedule changed. Despite these setbacks, the community grew stronger as the team 
members helped each other manage the project complexities. 

When it became evident that the final delivery schedule was not feasible, the instructor again revised the 
course requirements to support the successful completion of the course. Each team member developed a storyboard 
for his or her assigned module and the instructor lead a class discussion to reveal best practices. Each module team 
fully developed a prototype of at least a third of the instructional content and developed an informal peer formative 
evaluation of their module. The instructional strategies and graphic designs varied greatly among modules. This 
served as a powerful lessons learned for the team as it revealed the many ways which similar instructional content 
could be approached, and also the risks of creating a disjointed instructional course if similar practices are not 
employed across modules. The team celebrated the delivery of the prototypes to the client. 

The instructor negotiated with the client to extend the project into spring semester and seven of the original 
team members were able to return. An additional graduate student had heard about the course and persuaded the 
instructor to allow her to join. The student read the entire textbook over the winter break to gain domain specific 
knowledge, and in doing so demonstrated her commitment to the team. Over the break, the client reviewed the 
prototypes, selected favorite instructional elements, and shared his preferences with the team at the kick-off meeting. 

The instructor created a Revised Project Structure based on the number of students and a desire to allow 
each team member the opportunity to perform all instructional design functions (see Figure 2). Two modules were to 
be developed by individuals, while the other three were developed by teams of two. 

e-Learning Module 

Module 1: Module 2: Module 4: Module 5: Module 6: 

Introduction Design Planning Implementation Monitoring 



Figure 2: Revised Project Team Structure 


At the beginning of the spring semester the team worked together to simplify everything which would not 
sacrifice the quality of the client deliverable. This is evidenced by the deletion of the MS SharePoint site from the 
fall semester which had at the original request of the team included many layers of access rights intended to keep 
working documents private until the teams were ready to share. During the spring semester, a Google Site was 
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developed in which all team members could share digital assets, fonts, prototypes, and templates. Through 
notification settings, the Google Site also served as the team’s technical hotline and any member posting a question 
could confidently expect a response within minutes or hours. The site became the repository for all project files 
including the schedule, proposal, communication plan and archived technical support. The team members used Jing 
and Camtasia to record and share technical advice. The development of the project site was a testament to the team’s 
work experiences. One team member well trained in international instructional design project management quickly 
deployed the site for the team. Over time the team members became trusted friends invested in the team’s successful 
delivery of all modules rather than individuals focused on grades. 

In CA learners are required to be more engaged in the subject matter and for longer times which may pose 
some challenges. The time constraint was successfully overcome during the spring semester by meeting less 
frequently because the CA foundational content had already been developed. The instructor’s flexibility and comfort 
with ambiguity helped to prevent learners from becoming frustrated. The motivation of the service component led 
students to dedicate long hours to the project between meetings. While not required to do so, over the course of the 
semester the team chose to meet more often to share tips and maintain consistency while the instructor offered 
assistance. Multiple peer reviews served to identify errors and recognize exemplary practices that could be emulated 
in other modules. The team members began to model the instructor’s approach to feedback by asking each other 
questions rather than offering authoritative advice. 

Application of ID Principles in an Authentic Context 

To maintain nimble practices, the team employed a spiral instructional design model guided by iterative 
cycles (Cennamo & Kalk, 2005). Cennamo and Kalk’s (2005) spiral model combines five stages of instructional 
design (define, design, demonstrate, develop and deliver) in a non-linear fashion. As the project moved through the 
spiral, the team revisited key elements (learner, outcomes, activities, assessment and evaluation) and refined the 
approach with every rotation. This approach is similar to rapid prototyping models in that it led the team from a low 
fidelity to a high fidelity product. This process allowed the team to revisit previous stages while concurrently 
moving forward and several rounds of the entire design process were completed before final delivery. However, the 
team soon realized through observation and reflection that the process was not as tidy in practice. 

Recall that during the first semester, module groups were given the autonomy to develop their own themes 
using a basic framework provided by the client, however during the second semester this approach was revamped 
for a more successful approach of a unified theme throughout the entire e-Learning course. The strongest features of 
each prototype were combined into templates that provided the development teams with both a consistent structure 
across modules and flexibility within modules. Consistent introductory screens were designed by a team member 
and applied to all modules (see Figure 3). A navigation map containing learning objectives was included at the 
beginning of every module and content chunk to guide the learner through the journey. A comparison of the same 
screen from the prototype and the final version reflects the development of the team’s skills (see Figure 4). 

This experience provided a very good model of what was required to successfully manage an ID project in 
the real world. The instructor had the challenge of delivering an outcome to a client while at the same time ensuring 
that students were developing the necessary skillsets and all this had to be done in the confines of the allotted course 
period. It was obvious that to employ the CA framework one has to be flexible and comfortable with some level of 
chaos and ambiguity. 
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At this point you have learned the skills to navigate the first three 
phases of the life of a development project. Now, it is time to 
explore the project implementation phase. 


Are you ready to begin? 




Figure 3: Consistency of Introductory Screen Design 



£ PMD Pro 



Figure 4: Content Chunk Introductory Screen from Prototype and Final Delivery 


Practice was incorporated to reinforce learning and relevant feedback was provided. Learner practice 
prototypes were bland in design and reflected an intimidating testing environment (see Figure 5). As a result of 
guidance from the instructor, learner practice was redesigned with colorful graphics, friendly feedback and an 
opportunity to try again (see Figure 6). 


Test Yourself 

Question loll • Point Value: 10 


Four sources are identified during your brainstorming session. Can you align them to 
applicable community need? 


Normative Needs 


Go to a clean water service and see 
if there are long lines of people 


Comparative Needs 


PROPERTIES 
On passing, ‘fin: 
On fa ling,' Finish 


Figure 5: Learner Practice Prototype 
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Figure 6: Learner Practice Final Delivery 

Case studies contextualized for the international non-profit sector supported the learning and the navigation 
allowed the learners to explore rather than be restricted to a linear process. The instructional content was 
reorganized into a storyline intended to replicate real-life experience and model appropriate skills. The character 
driven story created a non-threatening environment where learners feel comfortable to express their ideas, take risks 
with demanding tasks, and thus enhancing knowledge transferring and problem solving skills (Jonassen & 
Hernandez-Serrano, 2002). Character names were selected to represent the diversity of learners and were 
consistently applied to all modules with the help of a Character Alignment document (see Figure 7). The main 
characters were novice project manager named Maria, and her mentor, Allison. 

Capitalizing on the Power of Team Diversity to Serve International Learners 

Diverse teams are more likely to possess a broader range of task-relevant knowledge and skills 
(Knippenberg, De Dreu, & Homan, 2004) which may lead to innovative solutions (Ancona & Caldwell, 1992). The 
team’s cultural diversity supported the development of predominantly international color schemes, images and 
language choices, while diversity in skill sets allowed for the formation of functional roles. 
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Role Within 
Our Storyline 

Name Within 
Our Storyline 

Name as found in 
the Media Assets 

Snapshot 

Director (supervisor of 
the Project Manager 
and team members) 

Allison 

Allison 


Project Manager 

Maria 



Team Member 

Bahati 

Lisa 


Team Member 

Cory 

c°" 

i 

Team Member 

Ian 

Ian 


Team Member 


Katie 

a* 

Team Member 

Paul 

Paul 

a 




Figure 7: Character Alignment 


Recall that during the first semester, the team members were assigned into functional and module sub- 
teams based on interest and known skills and did not fluctuate from those roles. However during the second 
semester a slightly different strategy was employed which greatly impacted team performance. Team members were 
assigned to specific module teams based on interest and as team member’s unique skills were revealed during the 
semester, new responsibilities were assigned. For example, a team member was responsible for making sure that the 
overall storyline flowed smoothly across modules while other team members skilled in graphic design were tasked 
with developing the course templates and usage requirements. Another team member designed a navigational map to 
identify module learning objectives which was embraced by the team and was given the responsibility of providing 
open source screen designs for incorporation into all modules. This strategy served the project well because the team 
members were novices and still discovering their unique talents but the flexibility of the structure allowed them to 
accept additional responsibilities for which they could successfully fulfill. 

Because the modules were to be translated into different languages, the team’s diverse language skills were 
an asset to the team. Non-native English speakers brought to the team’s attention the importance of making sure that 
text box sizes were big enough to support foreign language translation. Similarly, native English speakers became 
responsible for correcting grammar and ensuring consistency of the overall tone of the content. Three team members 
were hired by the client after the completion of the course to translate and localize the modules for Latin America, 
Haiti and Brazil. 


Evidence of Effectiveness 

To date, the seven modules have been released in four languages, for a total delivery of 28 modules to 80 of 
the client’s member organizations with 200,000 registered learners in 150 countries. Member organization learning 
management systems (LMS) are linked directly to the client’s LMS, Moodle. Modules are also accessible free of 
charge at http://www.lastmileleaming.org . In May 2013, one of the client’s member organizations shared its 
experience with these modules. While the sample is limited (79 learners), the results are favorable with a 97% PMD 
Pro 1 Certification Examination pass rate after taking the distance, blended learning course which utilizes the self- 
directed modules as a key component, versus a 75% pass rate historically achieved by member organizations 
through face-to-face instruction. The learners were trained at 5% of the cost of comparable face-to-face learner 
events, and greater diversity in gender and nationality were also reported. 

The success of these modules can be attributed to the explicit guidance of the instructor, team spirit of the 
class, and extremely cooperative client. Furthermore, after students completed the course, the client employed class 
members to develop remaining two e-learning modules, perform quality control, virtually manage volunteer ID 
teams, and perform foreign language translation. The CA continued as the students transitioned into the new roles 
with the client. The instructor helped to negotiate the student’s consulting contracts, address workload concerns, and 
provide project management advice. 
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Reflecting on the Cognitive Apprenticeship Experience 


Indeed the learning for this project was situated in an authentic context and tapped into the learners’ 
intrinsic motivations. The instructor served as co-creator and facilitator of the process and product, and a spirited 
community of practice blossomed as students worked together to accomplish the completion of five of the PMD Pro 
1 modules during the spring semester. The level of team spirit and energy among the students was amazing and can 
be attributed to the guidance and level of responsibility that the instructor gave the students as well as the service- 
learning nature of the project. 

In a project debriefing meeting after the successful delivery of the modules, students stated that, as a result 
of the experience, they believed they could confidently seek professional instructional design opportunities and 
speak intelligently about the process. Overall they learned how to effectively work with a client and manage the 
various components of an instructional design project; how to avoid panicking when things are not going according 
to plan, but to re-strategize and focus on the desired outcome. They had a greater understanding of the factors 
involved in deciding on the theories that informed the design of instruction. 

One team member quietly reflected to another that she believed the characters and storyline in the modules 
were actually a microcosm of their ID team. In both tales, the novice found success through the patient mentorship 
of a seasoned professional. 

While this paper describes the development of an international team of novice instructional designers, the 
instructor’s pedagogical approach of a service-learning CA may be readily applied to other contexts. In sharing this 
experience with the academic community, the team hopes to inspire other university programs to seek opportunities 
for student growth through service to others. 
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Introduction 

While mobile learning environments have some similar characteristics to computer based and web based 
systems, there are unique characteristics of mobile learning environments related to usability and functionality that 
will invariably affect the elements in the learning system. Gedik, Hanci-Karademirci, Kursun, and Cagiltay (2012) 
identified three categories of issues that are critical in the design of mobile learning: technical and technological 
issues related to the type and functions of the mobile device; curricular and pedagogical issues related to the purpose 
and strategies of the instruction; and management issues related to communication between design and development 
stakeholders. Given these unique characteristics, mobile technologies present an opportunity for scholars and 
practitioners to extend the application of instructional design theories and models to a mobile learning environment. 

The goal of this research-in-progress is to validate and extend the component display theory (CDT), a 
traditional instructional design (ID) theory. Using a formative research approach (Reigeluth & Frick, 1999), which 
focuses on improving design theory for instructional practices and processes, the CDT will be used to design a 
mobile application within the context of a fully online, faculty development workshop. In this paper, initial research 
and thoughts are offered on using the CDT. Specifically, a summary of the CDT, an example of its application, and 
initial thoughts from an instructional designer’s perspective on variables that will influence the design of the 
instruction are presented. 

The following three research questions guide this study: 

• How can the CDT be used to guide the design and development of instruction within a mobile learning 
environment? 

• What key processes are pertinent to translating ID plans into mobile learning lessons? 

• What are the challenges and issues in designing instruction for a mobile learning environment? 

Using CDT to Guide the ID Process 

The CDT (Merrill, 1994a) is a micro-level theory that prescribes presentational strategies for instruction 
based on the subject matter content (fact, concept, procedure, and principle) and the expected student performance 
(remember-general, remember-instance, use, and find). Through prescriptions, the goal of the CDT is to minimize 
the over and under generalization of information that may increase errors and difficulty in the transference of 
knowledge to the learner. Subsequently, the CDT prescriptions (1994b) have four parameters to guide the design of 
instructional strategies: 

• Primary presentation: discrete instructional presentation based on two dimensions, content mode 
(general or instance) and presentation mode (expository or inquisitory) 

• PPF content: elements that should be present in the primary presentation for consistency of content 

• Secondary presentation: information, such as mnemonics, feedback, and prerequisite information, 
used in addition to the primary presentation to enhance learning by helping the learner process 
information or by providing additional context. 

• Inter-display relationships: interrelationships between different presentation forms that will affect 
how learning will occur. The relationships include the difficulty and randomness of presentations as 
well as learner control in the instructional system. 
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The parameters offer explicit strategies and identification of instructional components that can make 
instruction more effective for knowledge transference. As a result, this framework makes the CDT attractive for 
designing instruction for a mobile learning environment, where the combination of discrete learning objects and 
learner/facilitator/content interactions is used for instruction in a learner-controlled environment. 

Example of Use 

For educators, applying copyright law and the principle of fair use in a digital learning environment can be 
complex and confusing. So, for faculty development at a two-campus, community college district, a lesson on 
copyright and fair use is being designed and will be delivered via a mobile application. The goal of this lesson is to 
explain the concept and principles of copyright law and how educators can identify and discriminate among various 
exemptions and limitations including, fair use, Digital Millennium Copyright Act (DMCA), The Technology, 
Education, and Copyright Harmonization Act (TEACH Act), and public domain. 

What follows is a design draft for the first module (Figure 1). The CDT is used to write the workshop 
objectives, based on the performance objective and subject content, and to identify the relevant instructional 
prescriptions. Based on the prescription, instructional strategies are developed, including how new information and 
examples are presented, what types of instructional support tools are needed, the types and quantity of practice and 
assessment items, and how feedback should be given to the learner. The complete prescription for the first module 
is presented and the instructional strategy summary identifies how each component is implemented. 

Module Objective: Learners will define copyright law and identify critical attributes. 

Instructional strategy summary: Learners are given a contextual situation in which the concept of copyright will 
be described. The definitions of copyright law as well as the key attributes that differentiate it from other 
intellectual property right, such as trademark and patents, are given (b). The definition is followed by an example 
that illustrates the critical attributions for copyright (c). Visual aids will be created to assist learners in remembering 
attributions (e, f). At least two different practice scenarios that provide the learner opportunities to identify 
attributes and recognize the definition will be offered (d, 1). The practice scenarios will be followed by the correct 
answer with explanation (g). 

Evaluation: After a short delay, correctly recall definitions and identify at least 4 attributes of copyright law. (j,k) 

Additional ID notes: Learners will control the pace and sequence of the presentation through navigational 
elements. 
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Figure 1: CDT Prescription Rule 5 


Performance Level: Remember Generality Content Type: Concept 


If S.E.T Attributes 

Then - Display Parameters 


Primary Presentation Form 
(a) EG + Eeg + IG.P + IG.P 

• An expository presentation consisting of a generality followed by an example, 
and a series of instances where the student responds by stating the definition. 


PPF Content 

(b) EG = Definition 

(c) Eeg (reference example) = Explanation 

(d) IG.P = State Definition 

• The presentation to the student consists of the name of the concept, 
superordinate class to which the concept belongs, a list of the attributes and 
values which distinguishes he class from coordinate classes. 

• The example presented to the student consists of specific objects, symbols, or 
events or representation which illustrates the attribute value of the definition. 

• For practice, give the name of concept, paraphrase the attributes and values of 
concept and the student will state the definition of the concept. 


Secondary Presentation Forms (SPFs) 

With EG: 

(e) Mnemonics = yes 

• The use of memory aids can be given to assist the learner in remembering. 
With Eeg: 

(f) Flelp =yes 

• Information added to the content to help the learner relate the instance to the 
generality 

With IG 

(g) Feedback =ca + h 

• The correct answer is given as feedback. An expository presentation of the 
problem is presented after the student attempt. 


Inter-display Relationships 

For all: 

(h) PPF isolation = yes 

(i) Learner control = yes 

• The primary presentation form is clearly separated and identified for the 
student by auditory or graphic conventions. 

• Learners have control over the pace, presentation forms, speed or any other 
learning parameters that can be controlled in the environment. 

For IG.P practice/performance 

(j) Response delay = short 

(k) Criterion = high 

(l) Number of items = at least two 

• Learner may have a short delay in responding to questions. 

• It is expected that learners’ accuracy in responding is high. 

• At least two instances are necessary for adequate instruction. 


Initial Thoughts 

Through initial work with the CDT, two areas of the model deserve future examination as to how it may 
affect the design and development of instruction. The first is the element of learner control, which determines 
whether the learner or the instructor/system makes decisions about the learning. Learner control is part of the inter- 
display relationship parameter in the CDT and is related to how the student determines the number of primary 
presentation instances to study, when to receive help, and other strategy decisions (Merrill, 1999a). 
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In discussing pedagogical affordance provided by mobile learning, Park (2011) adopts a hierarchical 
structure (Figure 2) of the mobile technologies that connects the formal and informal learning environment for 
learners. Level 1 focuses on individual learning whereas Level 4 focuses on collaborative learning. The hierarchy 
illustrations the range of mobile functions that can give learners the ability to adapt or extend the learning 
components, through various navigations, support aids, and interactions, to accommodate his/her own learning style 
and preference. For ID practitioners, decisions will need to be made on what types of mobile functions and features 
will facilitate appropriate control of the content, interactions between formal and informal networks for the learner, 
and how best to integrate control into the environment. 


Figure 2: Mobility hierarchy, sample applications, and technological affordances (Park, 2011) 


Mobility Hierarchy 

Sample Applications 

Technology Affordances 

Level 4 

Communication & 

• 

Real-time chat 

Communication intensive 


Collaboration 

• 

Annotations 

Group work 



• 

SMS (Simple Messaging System) 

Synchronous 



• 

Wireless email 

/ 

k 

Level 3 

Capturing & 
Integrating Data 

• 

Network database 




• 

Data collection/synthesis 





• 

Mobile library 

( Mobility ^ 

Level 2 

Flexible Physical 

• 

Local database 




Access 

• 

• 

Interactive prompting 
Just-in-time instruction 

\ 

t 

Level 1 

Productivity 

• 

Calendars 





• 

• 

Scheduling 
Contact information 

Asynchronous 
Individual work 



• 

Grading 

Content intensive 


The second element to further examine is the student, environmental, and task (S.E.T.) attributes that may 
put conditional controls on how prescription parameters are implemented. Based on the prescriptions, very little of 
the CDT model has been made conditional based on the S.E.T attributes (Merrill, 1994b). Mobile technology offers 
self-regulated learning opportunities, and the integration of informal and formal learning experiences. Explicit 
conditional controls based on S.E.T attributes may provide additional ways to support learners in a more 
personalized learning environment, specifically for the practice presentations, help, feedback, and learner control 
parameter elements. These two areas of learner control and S.E.T. attributes will be future examined as the 
instruction is developed, with expectations that the CDT model can be adapted to take advantage of mobile 
affordances and offer practitioners a framework to design and develop instruction for mobile learning environments. 
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Creativity, as a skill, crosses all disciplinary boundaries and is highly valued. Research has shown that 
creativity is three times as strong an indicator of lifetime achievement than intelligence [Plucker, 1999]. A large 
recent study by IBM also found that creativity is the leading skill needed by executives in today's world. The goal, 
with this course and this proposal is to increase the creativity of the entire University. 

Massive Open Online Courses are a growing trend in online and distance learning. The courses are free and 
open to the public and can attract 100,000 or more learners to each course. Several universities [ e.g. Berkeley, MIT] 
have begun to offer MOOCs on their own or through one of numerous hopeful ventures [ Udacity, Coursera, EdX ]. 
Few if any have developed a sustainable business model and only recently have they begun to charge a course 
certification fee. In most cases, no tuition is generated, no credit is offered, and learning is free to the learner. The 
percentage of students that complete the courses is small. The University of Minnesota has recently begun the effort 
to develop MOOCs and has recruited faculty willing to re-develop their courses to a MOOC format. 

In an atmosphere of increased accountability (and scrutiny of University budgets), one option is to focus 
our digital effort on the internal student body and on the potential to develop their creative skills for use in all fields. 

The concept is to develop a MOOC-like course that is offered to all entering University students for the 
summer prior to their freshman year. This would have an identified target audience of 5500 students and could be 
offered as part of their enrollment in the University. The course would be for three credits, as is the current face-to- 
face offering. Students who complete the course would be able to receive credit as part of their fall semester 
registration, which could be free within the tuition band, and at normal rates above the band. It is a SMOOCH, 
which stands for Semi-Massive Open Online Course, Here. 

The project involves a course titled Creative Problem Solving. It is a course in the College of Design that 
focuses on the development of creativity and problem solving skills that are needed by all students, not just 
designers. It is a required course for retail merchandising students, but substantial numbers of students from diverse 
majors enroll as well. Creativity, the ability to develop new ideas, applies to any discipline, with current students 
from honors, journalism, mechanical engineering, and the business school. 

Once operational, the course could be offered each year and could include transfer students, coordinate 
campuses, and continuing students at the University. It will also be offered as a regular course throughout the year in 
the MOOC format, and would include extensive cross-critique of student projects. It could also serve as a model to 
offer elements of a common curriculum or to add benefit to current University students. 

There are a number of benefits to this method. First of all, it advances the University's effort to improve the 
quality of undergraduate education and graduation rates. Students would be able to accelerate their education or 
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lighten the credit load of the shocking first semester. And, in a University charged with the development of new 
knowledge, all of our students could be more creative. 


Principal aspects of the course 

The course involves substantial use of problem based learning; a student project on creativity is due each 
session of the course. The course is structured as a series of challenges and problems to be solved, with discussion, 
critique and evaluation playing a major role in learning. It is not a lecture course where students are tested on the 
presented information; it is a very active learning environment. About 80% of course work is active participation by 
the participants including assignments, peer critique, discussion, and in lecture training exercises, and 20% is 
traditional information delivery and quizzes. 

Much of the course is currently on line in a course management system; all work for the term is collected 
online as project images and their written descriptions are easily collected and evaluated online. Quizzes and written 
exams can easily be presented in an online format. Discussion and critique could also take place online, with the 
current rubric based grading expanded to include peer evaluation. 

As part of a research regimen integrated into the course, the development of creativity has been carefully 
monitored since it's inception, using of the Torrance Tests of Creative Thinking. Developed at the University in the 
late 1950's, these tests provide a good insight into the most salient aspects of creative thinking. In my course, 
average measured increases in creativity range from 40% - 70% for the fluency portion of the tests; this is a 
measurement of the raw generation of new ideas and corresponds with a 1.5 standard deviation in normalized scores. 
In other words, the class on average increases from the 55 th percentile to the 92 th percentile in measured 
creativity. The methods developed in this course will be used to develop the SMOOCH course. [Note that 
comparable development in measured creativity does not occur independently through courses in the creative fields.] 

At the same time, additional access to and use of more advanced technology will make the course work 
exceptionally well, particularly in the area of peer evaluation and critique of online assignments. Most design based 
courses are hindered in their translation to a digital environment due to being unable to include student discussion 
and feedback to visual artifacts. Using rubric guided peer evaluation will improve the development of learning for 
both parties. 

Technologically and economically, the project seeks to be sustainable model for online education and one 
which provides access to University courses in an economically viable manner. We seek to effectively use our 
buildings over the summer; we also need to use other resources such as time, instructors, and digital resources to 
take advantage of that time as well. At the same time, during summers, place matters less than during the academic 
year; students in California or China can take this course as well as a student in Minneapolis. Or St. Paul. 

The course can be expanded using existing technology that is well tested, and the main activity of the 
course will occur over the summer when there is less server load than during the academic year. Teaching and 
graduate assistants are more available and less expensive over the summer making the operation of the course more 
cost effective. And incoming students' schedules are generally more flexible as well; not all will take the course due 
to choice, summer work, or other obligations; but it will be a good opportunity for many in the University. As a 
result, students will enter the University more engaged and ready to due generative work, to pursue research, and to 
develop their own ideas based on their coursework. 

The project is supported by the University's Provost and is currently under development. It is expected to 
be operational in June, 2014, and the first iteration will be complete in Fall, 2014. We are currently planning on an 
enrollment of 1000 students, primarily freshmen. 

From the inception of this project, the goal has been to transform the University, through helping students 
become more creative, through offering educational opportunities to students at a modest cost, and through 
improving the educational experience by helping more students complete their degree programs. The significant 
change is the use of MOOC technology to provide greater educational opportunity for University students for 
academic credit in a cost effective manner. 
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Abstract 

This paper is for university faculty in teacher education programs, inservice teachers, and others who 
provide training in technology integration for teachers. The focus of the paper is how to select various technologies 
for the classroom using instructional development tenets. Additionally, we will provide and explain a guide on 
technology tools selection that identifies advantages/disadvantages for each tool and their selection based on 
learning outcomes and instructional strategies. 


Introduction 

Most education programs in U.S. colleges offer courses on technology integration in the classroom. Often 
the focus on the functionality and use of various tools (Betrus, 2012). However, key to enhancing preservice 
teachers’ understanding is to provide guidance, not only on the capabilities of the technologies, but also application 
of basic instructional development (ID) principles (Davidson-Shivers & Hulon, 2013). This paper will extend this 
original discussion on ID and technology integration. 

Teachers and technology use 

Most teachers use technology tools in their classrooms on a daily basis. These technologies may be low- 
tech tools, such as pen and paper or worksheets, to high-tech tools, such as blogs and mobile devices (Davidson- 
Shivers & Hulon, 2013). Additionally, throughout the U.S., many PK-12 schools made financial investments in 
high-tech tools such as laptop computers, interactive whiteboards, e-readers, and so on, for teacher and student use 
(Angeli, & Valanides, 2005; Klieger, Ben-Hur, & Bar-Yossef, 2010). Some schools also allow students to bring 
their own mobile device and laptop computer for classroom activities. Such tools, just as with other resources, help 
teachers’ instructional delivery. Although these tools and initiatives support instructional implementation, they are 
only effective in the hands of a skilled and capable teacher (Angeli, & Valanides, 2005). Appropriate use of such 
technology, however, needs to be based on sound instructional design principles and skills rather than just on the 
features of the technology. 


Teacher education and technology training 

In most states, standard policy or regulation is for teacher education programs to require at least one course 
on technology use in the classroom (Betras, 2012; Gronseth et al. as cited in Foulger, Buss, Wetzel, & Lindsey, 
2012; Kleiner, Thomas, & Lewis as cited in Foulger et al., 2012). Currently, faculty in teacher education programs 
are tasked with preparing preservice teachers to know and use new and emerging technology tools. However, they 
might not be familiar with latest tools available (Betrus, 2012). 
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Common practices for selecting technology tools 

During most technology integration courses, preservice teachers learn about a variety of new, emerging 
technology tools. The course focus often emphasizes functions and capabilities of the technology and how to 
manipulate them adequately. In other words, students examine various tools at a basic level (i.e., what it can do and 
how to use them). The information tends to be very general and does not provide adequate information on how to 
select and critically evaluate tools for a given instructional or teaching situation (Betms, 2012; Kumar & Vigil, 
2011). Because of general information, transfer to actual classroom situations might not occur. 

This gap in transfer might exist because teacher education faculty depend on a single, and isolated, 
technology course to provide such training and may not model effective integration of technology in their own 
courses (Lim, Chai, & Churchill, 2011). It might be further exacerbated by the fact that even though classified as 
digital natives, not all college students are familiar with or have access to various technologies (Rasmussen, 
Davidson-Shivers, & Savenye, 2011). This lack of technology use makes it difficult for such students to come up to 
speed and maintain use of technology. 

Even when knowing about various tools, problems may occur when attempting to choose the appropriate 
one for a given instructional or learning situation. Carr-Chellman (2011) warned that selecting technology too early 
can be a mistake by novice designers; her warning could also be applied to preservice teachers. This difficulty in 
selection may be due, in part, to being enamored with the tool’s capabilities and not considering the context in which 
it will be used. It might also be due to little to no instruction on ID skills being provided (Davidson-Shivers & 

Hulon, 2013; Lim & Chai, 2008). 

Providing information on instructional development skills have been shown to be effective for teachers. For 
example, Angeli (2005) found that those teachers, both preservice and inservice, who are trained in instructional 
development (ID) skills, are more effective at designing and developing lessons that integrate technology. 
Additionally, teachers with ID skills tend to demonstrate significantly higher teaching skills than those without such 
skills (Aytekin, AbdulAziz, Barakat, & Abdurrahman, (2012). 

ID skills for aligning technology with outcomes and strategies 

Teachers are currently using technology on a daily basis to increase productivity in the classroom (Larson 
& Lockee, 2014). Some examples include using technology for communication (i.e. emails, class newsletters), 
grading student work, and using word processing or presentation software to create and share information with 
learners. However, when teachers are designing and developing instruction it is important that learning outcomes 
and the selection of technology tools are aligned. Instructional designers make technology selections by analyzing 
learning outcomes (i.e. goals and objectives), learners, instructional context and content (Davidson-Shivers & 
Rasmussen, 2006; Dick & Carey, 2009; Smith & Ragan, 2005). Based on the analysis findings, the instructional 
designer identifies the types of instructional strategies and delivery modes of the instruction. Technology or media 
selection can then be determined based on their given characteristics that best reflect the specific instructional 
strategies and delivery modes (Davidson-Shivers & Hulon, 2013). Davidson-Shivers and Hulon also suggested that 
selecting media also must take advantages and disadvantages of a given media into consideration. Using a slightly 
different approach, Larson and Lockee (2014) developed a four-step process to help instructional designers make 
technology selections. Their four-step examination process includes examining: 1) learners; 2) learning context, 3) 
instruction including identified strategies and pedagogy, and 4) depth and breadth of the content (Larson & Lockee, 
2014). Dick and Carey (2009) also suggested that the overall practicality of the tool may an impact the technology 
selection and, therefore, needs to be considered. 

Specific guidance on technology integration based on such ID principles needs to be provided for 
preservice teachers (Betms, 2012; Kumar & Vigil, 2011). One principle is that any given technology tool needs to 
be congruent with the specified learning outcomes and their corresponding instmctional strategies (Davidson- 
Shivers & Rasmussen, 2006; Van Eck, 2006). A second principle is that for instmctional or learning situations, it is 
the identified learning outcomes that drive the selection of appropriate instructional strategies and, in turn, the 
choice of these strategies drives the technology tool selection (Davidson-Shivers & Rasmussen, 2006; Smith & 
Ragan, 2005). However, Davidson-Shivers and Hulon (2013) suggested that these basic principles be taught to 
teacher education instructors so that they are able to identify and select technology for their own classroom use. 
Davidson-Shivers & Hulon also presented a table of about high and low technology that could be used in instruction. 

As an extension of Davidson-Shivers and Hulon’s (2013) work, our current guide focuses on learning 
outcomes* and corresponding instructional strategies for technology selection (see Table 1). Also included are the 
optimal low- and high-technology tools for each outcome are included and advantages and disadvantages of using 
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each tool within an instructional situation. The purpose of this guide is to help alleviate some of errors (i.e., selecting 
technology too early or making an inappropriate media choice), that novice users might make when selecting media. 


Table 1: A Guide to Select Technology Tools for Classroom Use. 


Learning 

Instructional 

Types of 

Tool 

Tool 

outcomes 

Strategies 

Technology Tool 

Advantages 

Disadvantages 

Verbal 

Direct 

LOW 

Ease of use 

If used too often 

information: 

instruction with 

Chalk- & white boards 

Affordable 

can be boring 

Declarative 

feedback 

PowerPoint slide show, 

Readily 


knowledge or 

Teacher / 

Paper worksheets. Drill 

available 


knowing what 

Student 

& practice 




presentations 

Board games or like 



Examples: 

Displays 

activities 



State all letters in 
the alphabet 

Practice 

HIGH 

Allow for 

Initial cost and 



Presentation software 

reusability when 

maintenance 

Name the 50 U.S. 


(i.e., SMART notebook. 

applicable 

Expense of 

state capitals 


Prezi, Slideshare, 


equipment & 



Google Docs, Keynote) 


software 

List Nine Events 
of Instruction 


E-games and activities 


Licensure of 
software & 
equipment for 
entire classroom 





use 

Intellectual 

Demonstrations 




Skills: 

Presentations 




Procedural 

Discovery 




knowledge or 
knowing how 

learning 




Discrimination: 

Discrim in ation : 

Discrimination: 

Discrimination: 

Discrim in ation : 

comparison of 

Comparison 

LOW 

Economical 


items by 

activities 

Worksheets to make 

Availability 

Lack of 

attributes 


comparisons of 

Can be 

immediate 



like/different items 

interesting while 

feedback 

Example: 


Match Games that 

learning 


Recognize 


require object 



different sounds 


comparisons 



as same or 



Economical 


different. 


HIGH 

Variety of 

May focus on 



e-games for matching 

games can be 

winning rather 

Match paint 


like items 

found online 

than learning 

colors 



Immediate 
feedback during 
games 

outcome 
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Intellectual 

Skills: 

Procedural 
knowledge or 
knowing how 

Concrete 
concepts: 
identify & name 
tangible items by 
commonalities 

Examples: 
Identify birds by 
their 

classification 

characteristics 

Defined 

concepts: classify 
abstractions by 
meanings 

Example: 

Classify 

democratic 

governments 

correctly 

Concepts: 
Discovery 
Learning 
Identifying or 
Classifying 
activities into 
like groups by 
tangible or 
intangible 

Concepts: 

LOW 

Manipulatives 
Visual displays of 
concepts 

HIGH 

Virtual manipulatives 
(ihttp://nlvm.usu.edu/en/ 
nav/vlibrary.html) 
Virtual manipulative 
apps on iTunes 

Concepts: 
Students are 
able to visually 
represent 
abstract 
concepts 

Inexpensive 
Digital natives 
are familiar with 
using mobile 
devices 

Concepts: 

Can be expensive 

Access to 
computers/mobile 
devices may not 
be equal for all 
students 

Intellectual 





Skills: 





Procedural 





knowledge or 





knowing how 





Rule-using: 

Rule-using: 

Rule-using: 

Rule-using: 

Rule-using: 

Apply lower- 

Direct- 

LOW 

Easy access 

Lack of 

order declarative 

Instruction 

Paper/pencil 

Inexpensive 

quantitative data 

& procedural 

or Demos. 

Class discussions 


with class 

knowledge 

Applications 

Partner talk 


discussions and 


Worked 



partner talk 

Examples: 

Examples 

HIGH 



Apply correct 

Question & 

Interactive response 

Immediate 

Requires training 

formula to solve 

Answer 

systems (i.e. 

feedback to 

for faculty 

algebra problems 

Activities 

“Clickers”) 

students and 

Can be expensive 


Practice 

Interactive polling 

teacher 

Mobile devices 

Write a short 


websites 


and additional 

story with correct 


(polleverywhere.com; 


hardware needed 

grammar and 


Socrative) 


Compatibility 

punctuation 




issues across 





campus or district 
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Intellectual 

Skills: 

Procedural 
knowledge or 
knowing how 

Higher-order (H- 
O) rules: 

assess complex 
problem 
situations & 
apply concepts/ 
lower order rules 
to solve 

Example: 

Generate an 
environmentally 
sound solution for 
rain run-offs into 
underground 
water sources 

Develop a lesson 
in your content 
area & for your 
students 

H-O rules: 

Problem-based 

learning 

Discussion 

Collaborative or 

team-based 

learning 

H-O rules: 

LOW 

Paper-based or video- 
based case studies 
In class assignments on 
complex problems 

HIGH 

Web-based case studies 
eGames 

H-O rules: 

Easily available 

Classroom 

allows 

for F2F team 

meetings 
Allows 
instructor to 
guide, direct & 
provide 
feedback to all 
students 
simultaneously 

Could be 
interactive and 
dynamic with 
author of case 

H-O rules: 

Feedback on 
proposed 
solutions might be 
incomplete or lack 
details about case. 

Time limitations 
for classroom 
activity 

Expense 
Accessibility to 
equipment & 
software 

Cognitive 

Lecture 

LOW 



strategies: 

Demonstration 

Paper/pencil 

Inexpensive 

Assisting students 

Monitor & 

on when, how & 

Word documents, data 

Readily 

in developing 

control own 

where 

or spread sheets 

available 

cognitive 

cognitive 

appropriate use 



strategies & their 

processes; 

Graphic 

HIGH 


knowing when 

learning 

organizers 

Mind mapping software 


and how to use is 

strategies; 


(i.e. Inspiration, 

Using mind- 

lengthy process 

metacognitive 


Mindomo, Microsoft 

mapping 


strategies 


Word or PowerPoint, 

software helps 




Apple Pages) 

students work 


Examples: 



toward digital 

Not always 

Adopt positive 



literacy 

available 

self-talk 





Self-check for 




Some faculty 

understanding of 




training needed to 

concepts 




become proficient 

presented 




with software 





tools 
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Attitude: 

Personal actions 
of choice to 
exhibit 

Examples: 

Choose to count 
to 10 before 
losing temper. 

Choose to start an 
exercise regime 
to improve health 

Role Playing 
Case studies 
Discussions 
Dramatizations 

LOW 

Paper scripts, books or 
readings 

HIGH 

Blogs (i.e. Blogger, 
Weebly, Wordpress, 
Edublogs) 

Podcasts (Audacity, 
GarageBand, AudioBoo, 
Moviemaker, iMovie) 
Videos (Animoto, 
Creaza, Khan Academy, 
Educreations, TED-Ed, 
jing) 

Inexpensive 
Readily 
available 
Can be 
developed by 
faculty 

Provide an 
outlet for 
students to 
express their 
thinking 

Storage space for 
paper scripts, 
books, and 
readings 

Monitoring the 
blogs for safety 
External 

equipment may be 
needed (i.e. 
microphones, 
webcams) 

Motor Skills: 


LOW 



Precision & 

Demonstration 

Live or Classroom 

Inexpensive 

Safety issues if 

coordination ov 

Endurance and 

demonstrations. 

Available 

using students for 

physical 

agility and speed 

Video demonstrations 


demonstrations 

movement skills 

training 

Physical practice on 





skills 



Examples: 





Execute a 


HIGH 



somersault; 


Digital storytelling 

Can use slow 

Assists, but not 

Dribble a 


software (Photostory, 

motion 

meant to alleviate 

basketball 


Animoto, Creaza, 

techniques to 

hands-on training 

correctly. 


Moviemaker, iMovie) 

identify critical 

Providing 

Perform a double 


Videos (i.e., Animoto, 

moves and form 

equipment & 

back flip 


Creaza, Khan Academy, 


software at 

correctly 


Educreations, TED-Ed, 


location in which 



jing) 


practice occurs 


Adapted from Clark & Sunhee, 2005; Davidson-Shivers & Hulon, 2013; Davidson-Shivers & Rasmussen, 2006. 


Summary 

Although technology selection is a part of developing and implementing effective instruction, it should be 
done in conjunction with presenting effective classroom application in a more specific manner. One way is to 
provide guidance to preservice teachers on selecting technology in coordination with aligning these tools with 
specific learning outcomes and instructional strategies. Such understanding could lead these teachers-in-training to 
not only make appropriate technology choices in congruence with their learning outcomes and instructional strategy 
choices, but also make their lessons and teaching more effective. Our guide might be a first step in assisting faculty, 
along with their students, in making decisions about the appropriate use of technology for any given instructional or 
teaching situation. 
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Abstract 

Online course enrollment is increasing in numbers; however, some instructors still think online course 
learning outcomes are inferior to campus education outcomes (Allen & Seaman, 2013). Even if this is true, the 
quality of learning outcomes depends more on course design than just on the course delivery medium. Despite 
growing distance education literature, a need still exists for empirically supported instructional design guidelines for 
effective distance education that promotes an in-depth understanding. In this paper, we examined interest, 
engagement, and learning approaches and their effects on achievement and provided instructional designers with 
guidelines to assist online course design. 


Introduction 

The 21 st century citizen will need to constantly improve his or her knowledge and skills to be successful in 
life. Educational institutions offer various means to accomplish such development of knowledge and skills. Online 
instruction is one of the means and has been increasing in popularity due to its anytime and anywhere nature (Parsad 
& Lewis, 2008). In addition to providing convenience, distance education practices are transforming constantly with 
the advent of new technologies and new educational approaches, such as massive online open courses (MOOCs) that 
have become popular recently. However, despite all advancements in the distance education field, a considerable 
body of instructors remains skeptical about learning effectiveness. Allen and Seaman (2013) reported that almost a 
fourth of the instructors nationwide thought online instruction outcomes were inferior to face-to-face instruction 
outcomes. The contention here is that the quality of the online course design is poor, rather than the mode of course 
delivery, online or otherwise. This argument is in line with what Bernard et al. (2004) found in their meta-analysis 
of empirical media comparison studies published from 1985 to the end of 2002. Considering findings supporting 
both sides of the debate and almost zero overall effect size, Bernard et al. concluded that the differences found 
between distance education and classroom counterparts were more likely to be associated with course design. 
Bernard et al. also noted that empirical literature often does not emphasize design features to inform course design. 
Thus, we endeavored to compile effective design guidelines with working examples to help instructional designers 
with the development of online courses. 

The main quality that makes design important in educational practice is that design paves the way to quality 
learning outcomes. Even though the ultimate purpose of education is to elicit long term learning outcomes, the 
problem of inert knowledge, knowledge that does not inform practice, is still prevalent in education (Reeves, 
McKenney, & Herrington, 2011; Whitehead, 1916). Students whose intent is to merely fulfill course requirements 
tend to engage with course content through surface level approaches, such as rote memorization. Surface level 
interactions often fail to create deep connections with the concepts that will in turn allow learners to transfer their 
knowledge to genuine situations (Beattie, Collins, & Mclnnes, 1997). As a result, knowledge remains inert, and 
learning outcomes become temporary. In order for students to comprehend the content, they should engage with the 
content meaningfully by making a good use of educational materials and their cognitive resources. Therefore, 
students should employ approaches to learning that promote deep understanding. 
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Learning Approaches 


Learning approaches define how a student perceives a task, and then, interacts with the task. There are 
three types of learning approaches: surface, deep, and achieving (Biggs, 2001). Each approach consists of a motive 
for studying and a study strategy. Students who employ a surface learning approach tend to aim for temporary 
learning, while students who employ a deep approach strive to achieve a deep understanding of content. Students 
with an achieving approach focus on obtaining higher grades by using time management strategies, and organizing 
their place of study. Even though achieving approach brings about better grades, it may be problematic because 
learning is not prioritized (Beattie et ah, 1997). High course grades do not always mean high-quality learning 
outcomes. For instance, a student with a surface approach may use low-level strategies and manage his time and 
effort efficiently to get the highest possible score without caring much about comprehension (Marton & Saljo, 

2005). In addition to personal motivations to study, course design may also alter students’ learning approaches. For 
example, a student may employ different learning approaches in alternative designs of a course on the same topic 
(Wilson & Fowler, 2005). Learning approaches are flexible and course design is of importance to direct students to 
engage in deep learning (Biggs, 2001; Garrison & Cleveland-Innes, 2005; Marton & Saljo, 2005). Thus, we need to 
address ways in which design communicates to students and encourage students to use a deep approach to learning. 

Marton and Saljo (1976a, 1976b), Svensson (1977), and Fransson (1977) conducted as series of 
phenomenological studies to examine qualitative differences in students’ learning outcomes and design components 
that lead them to comprehend the content of a reading passage. In their initial analysis, they found that students who 
understand the authors’ point intentionally made connections among sections, monitored their understanding, and 
asked thought provoking questions to themselves. However, students who fail to understand the main ideas tried to 
memorize as much as possible to succeed in the test that would follow. It was clear that the former group adopted a 
deep approach to learning, while the latter engaged with the task at the surface level. In follow-up studies, Marton 
and Saljo manipulated task demands to encourage surface learners to use strategies that deep learners tend to 
employ. Marton and Saljo conducted a study where they gave students exact questions that deep learners usually ask 
themselves while studying, and in another they provided sample, but not exact, questions likely to be asked in the 
test. Both studies reached limited success: some surface learners adapted their strategies to the task while some kept 
using a surface approach. After these efforts, Marton and Saljo concluded that even though students adapt their 
approaches to learning depending on task demands, merely manipulating task was not enough to empose a deep 
approach to students. Rather, instruction should also establish and support learners’ interest in the content to create a 
desire for understanding. 

As Marton, Saljo and their colleagues’ work indicates, the levels of interest and engagement determine the 
learning approach a student use. We provide a relationship model for interest, engagement and learning approaches 
here. These factors are interwoven, and thus, a change in one factor affects the others (see Figure 1). Although 
cognitive processes happen mostly at a personal level, instructional practices can guide students in appropriate 
methods and alleviate problems that defeat cognitive resources. Instructors can enhance the quality of online 
learning outcomes by integrating design components that promote students’ interest, engagement, and use of deep 
learning strategies (Sankaran & Bui, 2001). In the following sections, we will briefly explain the concepts of interest 
and engagement and their relationships with learning outcomes. 



Learning Outcomes 
(Surface Deep) 


Figure 1: Relationship Model for Interest, Engagement, and Learning Approaches 
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Interest 


Learners’ attitudes towards educational content are affected by motivational factors, one of which is interest 
(Hidi & Renninger, 2006). According to Hidi and Renninger (2006), interest is an individual’s situational interaction or 
inclination to interact with a specific content or object over time. This definition covers two important characteristics of 
interest: content-specificity and development. Interest is content specific, that is, students differ in their topics of 
interest (Hidi, 2006; Hidi & Renninger, 2006; Schiefele, 1991). For example, people may be interested in specific 
subjects, such as history or mathematics. Additionally, there are also variations in the topics of interest even in the same 
subject matter. For example, a student may be interested in plant biology but not be interested in animal biology. 
Moreover, interest is not a static concept. Cognitive and affective factors may result in the emergence and development 
of interest in a specific topic (Hidi & Renninger, 2006; Krapp, 2002; Schiefele, 1991, 2009). 

Interest researchers divide interest into two main categories: situational and individual. Situational interest 
is a state component, which is triggered by situational factors readily available in the learning environment (Hidi, 
2006; Hidi & Renninger, 2006; Krapp, 2007; Krapp & Prenzel, 2011; Renninger & Hidi, 2011; Schraw & Lehman, 
2001). Also situational interest is relatively short in duration and loses its effect if not maintained through further 
engagement. On the other hand, individual interest is the trait component of interest, and it is derived from personal 
experiences and values. Individual interest is relatively stable in comparison to situational interest. Thus, 
environmental factors have fewer effects on its development. Even though individual interest and situational interest 
have different theoretical features, they are intertwined, and transitions between these types of interest are possible 
(Dohn, Madsen, & Malte, 2009; Hidi, 2006; Hidi & Renninger, 2006; Renninger & Hidi, 2011). 

According to Hidi and Renninger’s (2006) four-phase model, interest develops during the phases of 
triggered situational interest, maintained situational interest, emerging individual interest, and well-developed 
individual interest, respectively. Instructors should not assume that their students are either are interested or not. 
Interest is open to development and student-content interaction plays an important role in its development (Hidi & 
Renninger, 2006; Renninger & Hidi, 2011). Through repeated engagement and the support of environmental factors, 
a person’s interest may develop and move to a higher level, while in the absence of support and engagement, interest 
may remain constant or diminish. (Hidi & Renninger, 2006; Renninger & Hidi, 2011). Individual and environmental 
differences in the learning experience determine the strength of the phases. Situational interest is especially 
important to educational practice because not all students have genuine interest in course content, and situational 
interest is the concept that guides instructors in fostering interest (Ainley, 2006; Hidi, 2001; Krapp & Prenzel, 2011). 
Content presentation and activities, for example, can help to trigger the interest of students who have little or no 
prior knowledge about the topic (Dohn et al., 2009; Durik & Harackiewicz, 2007). As a result, this triggered interest 
can be maintained and improved through repeated engagement. 

Interest alone, however, does not directly affect learning outcomes. Rather, it influences mediating 
variables and the learning process (Krapp, 2007). Students with interest pay more attention to course content, 
carefully consider details, and strive to use additional resources. This sort of activities lead students to build stronger 
connections between the content and what they already know. Interest also influences goals: students with a high 
level of interest set more advanced goals (Hidi & Renninger, 2006). Such students tend to regulate their own 
learning process and effort to establish desired goals. Instructors can make use of interest in course design to make 
content more valuable to students. When students’ valuation of information is increased, learning becomes more 
than mere knowledge transition and students have more reasons to cope with challenges. Students tend to actively 
engage in learning course content in which they are interested, therefore using deeper approaches to learning 
(Schiefele, 1991). Interest, then, becomes a foundational component for deep learning approach. 

Engagement 

The companion of interest in deep learning approach is engagement. Engagement refers to one’s cognitive 
and physical participations in learning activities. Engagement is vital to knowledge acquisition and the development 
of interest. Three prominent kinds of engagement are referenced in the engagement literature: behavioral 
engagement, emotional engagement, and cognitive engagement (Fredricks, Blumenfeld, & Paris, 2004; Reschly & 
Christenson, 2012). Behavioral engagement refers to observable aspects of engagement that range from simply 
following a teacher’s directives to actions that require the student’s initiative (i.e. asking questions). Emotional 
engagement refers to students’ feelings about learning components, such as teacher, topic, and classroom climate. 
Cognitive engagement relates to being purposeful and strategic in the process of learning. 

External factors, such as family, outside community, culture, and educational context affect engagement 
(Fredricks et al., 2004). Although these factors are all important, we need to focus on the educational context. 
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especially the classroom context, because of its strong relationships to interest, learning approaches, and 
achievement, and because educators have direct control over the classroom context. The classroom context consists 
of teacher support, peers, classroom structure, autonomy support, and task characteristics. Instructors can increase 
student engagement by manipulating these factors (Fredricks et ah, 2004). If student engagement is maintained over 
time, interest is triggered, sustained, and enhanced (Hidi & Renninger, 2006). The quality of engagement correlates 
to the quality of learning outcomes; students need to cognitively engage with activities in order to comprehend the 
content in greater detail. Students’ engagement can be increased by providing structure, autonomy, optimal 
challenges, and peer interaction (Fredricks et ah, 2004). Engagement becomes a necessary component for effective 
course design. 


Obstacles to Deep Learning 

Clearly, interest, engagement, and learning approaches affect the quality of online learning outcomes. 
Instructors should support all three of these factors for sustainable results. The education literature suggests that 
learners often fail to learn deeply because of lack of interest, limited engagement, and ineffective use of learning 
strategies (Floyd, Harrington, & Santiago, 2009; Jang, 2008; Kyndt, Dochy, Struyven, & Cascallar, 2011; Lim & 
Morris, 2009). Before discussing methods guidelines to support learning, it is necessary to mention problems that 
learners encounter in online higher education. 

The first obstacle is limited interaction among instructors and students, which is quite common in online 
learning. Students, especially those who are having their first online experience, may be reluctant to interact with the 
instructor and peers because in most cases, an online environment might not provide them with much information 
about their classmates and instructor(s). The amount of student interaction may also be influenced by whether a course 
is synchronous or asynchronous. Asynchronous courses can be accessed from any place at any time, while 
synchronous courses can be accessed at any place, but during the same time. In online courses with synchronous 
components, students sometimes only see a list of names and hear the voice of the speaker, either the instructor or 
students. Asynchronous courses often do not include even this limited interaction (Offir, Lev, & Bezalel, 2008). Due to 
limited socialization, students may choose to remain distant from learning activities by, for example, not participating 
in discussion unless required. While students’ needs for a sense of community may differ considerably (Exter, 
Korkmaz, Harlin, & Bichelmeyer, 2009), students who need a strong sense of community may feel isolated if such a 
community does not exist (Drouin & Vartanian, 2010; Rovai, 2007). Moreover, students, in general, appreciate the 
kind of activities that can establish a professional learning community (Drouin & Vartanian, 2010). 

The second problem is that students may not know the importance of the class. The course syllabus may 
mention general course content, but the information on how content will help students in their personal and 
professional lives is generally missing. For example, a graphic design course’s syllabus may list the topic that will 
be covered during the semester, which is usually enough for those who already have interest in design. However, 
students from different focus areas who may want to learn design may not be able to discern the relevance of course 
content, which could lead them to undervalue the course and disengage (Rovai, 2007). The same situation is also 
true for students who take online courses as electives. 

The third problem is time constraints (Song, Singleton, Hill, & Koh, 2004). Most college students take 
three or more courses in a semester and clearly have other activities to pursue as well. For example, working 
students have to spend time on both their jobs and their education; also, full-time students take more courses. This 
abundance of tasks makes time management skills crucial in higher education, especially in distance higher 
education (Song et al., 2004). If students lack these skills, they may feel overwhelmed and struggle to focus on one 
task at a time; as a result, students may submit superficial work. Thus, time management issues often lead to low- 
quality outcomes and short-term learning or so-called surface learning. 

The forth problem is the misinterpretation of course requirements. Even when the instructor gives students 
a detailed list of the assignments in advance, students may overlook the list and misinterpret what they need to do 
(Rovai, 2007; Song et al., 2004). This situation can affect students’ time management and strategy use (Song et al., 
2004). Students, for example, may underestimate a seemingly easy task and allocate little time for it. When students 
start working on the assignment, they realize it requires more resources, such as time and effort, than they imagined, 
which may cause stress and result in employment of a surface approach. Moreover, students may misunderstand the 
requirements and submit low-quality work. 

These obstacles, as a result, can limit online students’ productivity, cause negative feelings, such as 
frustration and helplessness, and, in turn, low-quality learning outcomes or even dropouts. The way to overcome 
such obstacles is for instructors to be aware of the challenges learners often face in an online environment in a 
proactive fashion and assure that course design provides opportunities to engage in a deep approach to learning. 
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Guidelines to Support Interest, Engagement, and Learning Approaches in Online Learning 


Education literature provides us with various methods to promote interest, engagement and learning 
approaches, and to overcome the limitations of online learning environments, such as limited interaction with peers 
and instructors, low perceived course value, time management issues, and unclear expectations. Although these 
guidelines increase the time that is spent on course preparation, they do not consume much instructional time. 
Instructors can apply all of these guidelines in one course and adjust their designs in each offering as needed. 
Frequently Assess and Address Students’ Needs 

Needs assessment is a crucial process in instructional design. Instructors should know their audience well 
in order to design optimal online learning environments (Herbert, 2007; Lim & Morris, 2009). Teachers and 
instructional designers need to identify the reasons students are taking the course, their current knowledge about the 
topic, and their understanding about the way the topic has been developed. Instructors should adjust course design 
according to these assessment results. For example, the emphasis of the course value, such as instrumental task 
value, can be embedded in the course for the students who have little interest in learning the course material except 
to earn credit hours for their graduation. When students have improved perceptions of a course’s value, their 
interest, engagement, and use of a deep learning approach will also increase (Floyd et al., 2009; Jang, 2008; Rovai, 
2007). 

Instructors should make sure their audience is knowledgeable about metacognitive strategies and time 
management strategies in order to accomplish course objectives. If students are not knowledgeable, instructors need 
to provide them with strategy training (Rovai, Wighting, & Liu, 2005). Online courses, especially ones at the 
graduate school level, accommodate students with various skill sets and technical knowledge. Instructors can 
provide prerecorded tutorials and support students in overcoming challenges that are not directly related to course 
content (Ascough, 2007; Song et al., 2004), so extraneous cognitive load can be prevented (Cifuentes, Alvarez, & 
Edwards, 2011), and students can allocate more cognitive resources for course objectives. 

Support Autonomy and Provide Structure 

Instructors should encourage autonomy based on student interest and need by providing optimal challenges 
and choices for learning objectives and activities. The structure of the course must be aligned with autonomy; clear 
expectations, explicit directions, and immediate feedback are essential (Hara & Kling, 1999; Rovai, 2007; Song et al., 
2004). When the structure and autonomy support are well balanced, student engagement improves (Jang, 2008; Jang, 
Reeve, & Deci, 2010). Instructors should provide students with clear, but not limiting, guidelines. For instance, a 
scaffolding that points out the features of a good research topic and gives examples of what is researchable and what is 
not could be a good supplement of a research activity. Detailed rubrics and sample works done by former students can 
also be used to clarify expectations. By doing so, the instructor both supports students' feeling of autonomy and 
prevents students from misinterpreting the activity or getting lost during the process. When designing activities, it is 
important to keep in mind that every student has different needs for autonomy and structure. Therefore, flexibility in 
assignments is essential for the support of both experienced and inexperienced students. 

Optimize Task Difficulty 

Instructors need to design learning tasks in ways that balance task complexity and difficulty. Students’ 
perceptions of tasks affect their selection of learning approaches (Biggs, 2001; Kyndt et al., 2011). Therefore, 
instructors should provide optimal challenges for learners. When challenging students, instructors should be careful 
not to overload students. Excessively challenging tasks or a high workload can cause extraneous cognitive load and 
force students to use a surface-level approach. For example, interested students may use rote memorization due to 
the time constraints of an activity. Instructors should also consider their students’ overall workload. Students 
generally take more than one course each semester, so assignments should be given at a level that can be balanced 
with their other course work. With a high overall workload, students may feel overwhelmed and have difficulty 
employing a deep approach. Case and Gunstone (2003) investigated the factors that affect students’ learning 
approaches in a qualitative study. The analysis of interviews and journal tasks revealed that students’ selection of 
learning approaches were limited by time pressure. 

Organize Learning Materials 

Materials should be presented in ways that help students to realize the relationship between their prior 
knowledge and what they are learning, so students can synthesize the information. Well-organized materials help 
students choose and apply appropriate cognitive strategies more effectively. Disorganized materials increase 
extraneous cognitive load, which causes confusion and reduces the efficiency of strategy use. In order to reduce 
external load, instructors can use diagrams instead of text heavy slides. Diagrams encourage students to see complex 
relationships at a glance. When supported with concise explanations, diagrams yield better learning outcomes 
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(Moreno & Mayer, 1999). Nonetheless, designers should not forget that overly complex diagrams could result in an 
adverse effect. In the study of Soric and Palekcic (2009), the successful students reported that they used visuals to 
aid their learning and make recall easier. As this example indicates, instructors can also guide students in organizing 
information in personally more meaningful ways, so they can synthesize and recall information effectively. 

Provide opportunities to create a learning community 

Peer interaction affects the quality of discourse, students’ benefit from learning activities, and motivation to 
cope with emerging challenges (Rovai, 2007; Swan, 2002). To get students interacted with each other, instructors can 
employ online discussion boards and various Web 2.0 tools. Nevertheless, when students are forced to use discussion 
boards, such as assignments in which every student should make at least 5 comments, the quality of discourse may 
suffer. Indeed, students should be encouraged to interact with each other, but interaction events should be meaningful 
and support the accomplishment of the learning goals, rather than becoming busy work just to be able to say there was 
some kind of student input (Rovai, 2007). Instructors should monitor online interactions as a contributing facilitator and 
should not let any learner dominate or abandon the discussions (Rovai, 2007). Junco, Heiberger, and Loken (201 1) 
gave a good example of motivating students to actively engage in discussions by using an online tool. In their study, 
they used Twitter to communicate. The instructors posted a starting question and asked all students to respond to that 
question individually. Everybody could read others’ responses and that openness triggered an exchange of ideas among 
students. As a result, students started to build strong relationships with their peers, and collaborated more. As this 
example indicates, students are inclined to interact. What instructors should do is to initiate these interactions in a 
voluntary, not forced, fashion and involve themselves in online interactions. In another study, Garrison and Cleveland- 
Innes (2005) compared four different online course designs and their effects on learning approaches. They found that 
the course designs with activities that required reflection and thought resulted in increased use of deep approach during 
the semester. Jung, Choi, Lim, and Leem (2002) investigated the effects of collaborative interaction (e.g., working in 
groups, exchanging ideas) and social interaction promoted by the instructor. The results indicated that the social and 
collaborative interaction groups engaged more in posting activities than the control group. Conrad and Donaldson 
(2011) posited a step-by-step approach in community building. They suggested starting with icebreaker activities to 
establish social-emotional communication and continuing with peer-reviewed individual assignments; next, work in 
dyads; finally large group projects. The idea is that by gradually approaching to large group projects, students have 
more opportunities to experience each others’ strengths and working styles in a comfortable fashion, so students can 
make an informed decision when forming groups. This approach to group formation supports students’ engagement by 
preventing conflicts that may occur within student groups. 

Conclusion 

Distance education, in general and online education in particular, is continuing to grow as an instructional 
delivery medium. Every new technology or refinement of current technologies makes online delivery more flexible 
and accessible. The recent applications of MOOCs are just an example of what can be done in distance education. 
However, instructor and course designers should give attention to design features and sound pedagogical approaches 
in addition to providing access and flexibility. Without such attention, we deviate from the main purpose of 
instruction, which is to facilitate the construction of knowledge. High-quality knowledge construction is most often 
achieved through deep learning opportunities. Extant literature states that interest, engagement, and learning 
approaches are the keys to deep learning. A conceptual model that shows the interplay of the concepts is proposed 
here, and guidelines to promote them are provided. Instructors can enrich online learning experience and foster the 
use of a deeper learning approach by monitoring students’ needs, supporting both autonomy and structure, 
organizing materials meaningfully, and establishing an online learning community. 
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Abstract 

Research on the efficacy of using visual diagramming tools to facilitate argument analysis has been mixed. 
One reason for the mixed findings is that no empirical studies have yet been conducted to formally identify the 
sequential steps (and underlying reasoning processes) students use when constructing argument diagrams. This 
study tested an innovative set of visual analytic tools/methods to record, sequentially analyze, visualize, and 
compare sequential patterns in argument diagramming actions performed by graduate-level students that created 
high (n= 12) and low ( n = 12) quality argument diagrams. A comparison of the transitional state diagrams depicting 
the patterns in actions sequences performed by high versus low performing students revealed five unique action 
sequences (out of nine total observed patterns) used by high performing students. These unique actions identify the 
diagramming processes that can be promoted to help students create more accurate argument diagrams and to 
achieve a better understanding of complex arguments. 

Introduction 

Critical thinking is an important skill that enables one to accurately reason and judge information and 
become lifelong learners for the 21 st century. It has been defined as ‘the art of analyzing and evaluating thinking 
with a view to improving it’ (Paul & Elder, 2001) and an intellectual standard that includes clarity, accuracy, 
precision, relevance, depth, logic, and breadth (Mclean, 2005). However, recent research suggests that many college 
students fail to develop critical thinking skills to the extent that they can effectively use them (Kuhn, 1991). To 
address this problem, various methods have been used in higher education to teach students the skills of 
argumentation and argument analysis across many disciplines. Argument analysis is the study of logical 
relationships between propositions presented in an argument (which can be mutually supporting or opposing 
opinions/claims) in order to reason through premises to reach a conclusion. In argument analysis, students identify 
the functional roles of each proposition (i.e., conclusion, premise, co-premise, counterargument), analyze the 
hierarchical relationship among propositions (i.e., levels of premise), and evaluate the quality and line of reasoning. 
This process helps students to correctly judge the quality and identify flaws within an argument and help students to 
make well-reasoned decisions. 

Given that arguments are often complex and ill-structured, argument mapping software have been 
developed to help students draw diagrams and map out the hierarchical relationships that link minor to major 
premises (Braak et al., 2006). Diagramming software like Belvedere (De Neys, 2006), Rationale (van Gelder, 

2007), and jMAP (Jeong, 2010) enable students to draw, position, and link multiple nodes to map out, represent, and 
convey complex hierarchical relationships between premises. Yet, a critical review of the research on argument 
diagramming and visualizing tools revealed that the majority of the studies found no significant differences (Braak 
et ah, 2006) and were flawed in experimental design. Furthermore, students’ maps often vary widely in accuracy 
regardless of the instructional intervention (Scavarda et ah, 2006). Ruiz-Primo & Shavelson’s (1996) review of the 
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research lead to the conclusion that students’ maps should not be used to assess students’ learning until students’ 
facility, prior knowledge, and processes used to create the maps are thoroughly examined. 

Because little if any research has explicitly modeled and identified the mapping processes that 
enable/inhibit students’ to accurately analyze/understand complex arguments, Jeong (2010) just very recently added 
to the jMAP software application the ability to chronologically log every action students perform while constructing 
complex diagrams. This type of data can be sequentially analyzed to visualize, reveal, and identify the reasoning 
processes that produce high and low quality diagrams. By using this latest version of jMAP, this study addressed the 
following questions: 

1. Which sequential patterns in students’ argument diagramming processes produce more versus less accurate 
argument diagrams? 

2. What particular techniques in the data manipulation, partitioning, processing, and analysis reveals patterns 
of action sequences that best distinguishes the students that produce the most vs. least accurate argument 
diagrams? 


Method 

The participants were 1 7 graduate students enrolled in an online graduate-level course on computer- 
supported collaborative learning at a large Southeastern university. Students were instructed to review arguments 
presented in an online debate (taken from another course) to support and oppose the claim: “One’s choice of media 
significantly affects learning”. They were instructed to reflect on how the concurrent use of argument diagrams 
might facilitate the debate. After viewing a short video on how to use jMAP, each student downloaded one jMAP 
file to diagram the supporting arguments and another to diagram the opposing arguments. In both cases, students 
were presented an initial screen (Figure 1) containing up to 15 nodes (specified in advance by the instructor) 
representing the main claim, supporting premises, and/or opposing premises. Students were instructed to begin by 
placing the main claim at the top of the screen. 

Each students’ map was scored against the instructor’s map (Figure 2) across three criteria: a) percentage of 
links in the student’s map that are in the instructor’s map; b) number of nodes that were correctly identified as a root 
cause (node with no arrows pointing outward, with at least one arrow pointing inwards); and c) number of chained 
links stemming directly from each correctly identified root cause up to the main claim. Data from the top 6 and 
bottom 6 maps of the supporting arguments and from the top 6 and bottom 6 maps of the opposing arguments were 
selected to produce a total of 12 best and 12 worst maps. The logged actions recorded in the top 12 maps were 
collated and reduced down into six categories (Table 1). The same process was repeated with the bottom 12 maps. 
The first data set was imported into the Discussion Analysis Tool (DAT) software to produce the left transitional 
state diagram in Figure 3, while the second data set was used to produce the right state diagram. 



Figure 1. Student’s initial screen with premises randomly arranged on screen 
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Figure 2. Instructor's diagram visually and quantitatively compared with student x's diagram. 


Table 1. Codes assigned to each action a student performed injMAP 


Category 

Code 

Definition 

LINK 

ADDR 

added new link pointing to the right 


ADDL 

added new link pointing to the left 


ADDU 

added new link pointing up 


ADDD 

added new link pointing down 


LK2 

attached link to the affected node 

RELINK 

RLK1 

redirected the existing link to a new causal node 


RLK2 

redirected the existing link to a new affected node 

- 

ULK1 

detached the beginning tail of the link 

- 

ULK2 

detached the end of the link 

ATTR 

ATT- 

changed link to color red to convey a negative or inverse relationship 


ATT+ 

changed link to the color black to convey a positive relationship 


ATT2L 

changed link to low level of impact 


ATT2M 

changed link to moderate level of impact 


ATT2H 

changed link to high level of impact 

DEL 

DEL 

deleted the link 

MOVE 

MS 

moved a node (which was the same node as the last moved node) 


MDn 

moved node to the north of the previously moved node 


MDne 

moved node to the NE of the previously moved node 


MDe 

moved node to the East of the previously moved node 


MDse 

moved node to the SE of the previously moved node 


MDse 

moved node to the South of the previously moved node 


MDsw 

moved node to the SW of the previously moved node 


MDw 

moved node to the West of the previously moved node 


MDnw 

moved node to the NW of the previously moved node 

COMM 

COM 

added comment to link to explain how node influences affectednode 


CREV 

revised the existing comment on the given link 
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Findings & Discussion 


The state diagrams (Figure 3) show that top scorers exhibited five patterns observed only among top 
scorers. These five patterns, which identify action sequences that explain how top scorers produce accurate 
diagrams, can be incorporated into a research-based set of guidelines on how to construct argument diagrams. For 
example, the left state diagram shows that when top scorers deleted a link, they were most likely (44%) to follow 
that action by adding a new link between nodes than bottom scorers. When they specified the attribute of the link, 
they were most likely (35%) to follow that action by adding a new link. Overall, the differences between the two 
state diagrams suggest that the following action sequences can help students construct more accurate diagrams: 

Del cteLink - ^ Add Link, Attribute - ^ Add Link, Attribute-^ Attribute, Relink Effect (move the head of the arrow to 
point to another affected node)^Relink Effect, and Relink Effect-> Relink Cause (move tail of arrow to point to 
another causal node). Other analytical techniques, findings, more detailed interpretation of the findings, and 
limitations of the techniques will be presented at the time of presentation, along with discussion of the instructional 
implications and directions for future research. 
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836>827 
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LINK 
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Bottom 12 Diagrams 


Note : The thickness of the arrows conveys transitional probabilities; black/gray colored arrows identify probabilities that are and 
are not significantly greater than expectedbased on z-score tests performed in the DAT software; first and second numerical 
value displayed in each node identifies the number of times the given action was perfonned and the number of events that 
followed the given action; the size of the glow emanating from each node conveys the number of times the action was performed. 


Figure 3. State diagrams of processes used to produce the top 12 versus bottom 12 diagrams. 


Directions for future research 


Although the findings are not conclusive, this preliminary study used sequential analysis to gain insights 
into the types of action sequences that students can perform when diagramming arguments in order to help students 
better understand complex arguments. Some issues that require further research are: a) replicate this study with a 
larger sample; b) integrate the target action sequences directly into argument mapping software interface so that the 
target processes can be tested in a controlled experimental; c) test the efficacy of the target processes while 
controlling for individual differences in students’ prior knowledge/understanding; d) examine to what extent the 
target processes are effective across different domains or topics that are or are not temporal in nature; and e) test 
other criterion measures for assessing accuracy. 


455 


Development of Instructional Materials Model Based on 
Inquiry-Transactional Approach 

Khoirawati 

fairuzanidf@yahoo.com; +62 813 21961314 
Sekolah Menengah Kejuruan (SMK) Negeri 1 Pagar Alam 
Jalan Raya Pagar Alam - Lahat KM 22 Pagar Alam, Sumatera Selatan 31551, Indonesia 

Descriptors for use in the index: inquiry transactional, development, design instructional. 


Abstract 

The purpose of this study was to develop instructional materials model. The process is develop a 
conceptual model of instructional materials, procedural modeling in the design instructional through systematic 
measures, effective, and efficient, and produce physical models for the new of instructional materials. This study 
uses research and development that combines two methods. The first method, taken from the initial step of research 
and development by Borg & Gall (1979) and the second method taken from the Systematic Design of Instructional 
Dick, Carey, & Carey (2009). The Systematic Design was adopted by Borg, Gall,& Gall (2007) as research and 
development method. The instructional materials developed based on inquiry-transactional approach. The final 
result will be the physical model of instructional materials based on inquiry-transactional approach. Those are 
printed and interactive CD instructional materials plus student’s and teacher’s guides. 

Introduction 

One of the subsystems in the class lesson is the availability of instructional materials and teaching methods. 
Instructional materials can be in form of books or printed materials, instructional CDs, online material, television 
broadcast material, radio broadcast material, or combination of these various forms. When seen from how the 
presentation of the content, instructional materials can use specific method or approach. The definition of 
instructional method is the way of teachers in delivering instructional materials so that the instructional objectives 
achieved well. 

Until now the availability of instructional materials such as text books for Vocational High School students 
especially for Educational of Islam Relegion (PAI) subjects is still limited. This limitation is not just about numbers 
but also especially regarding the book quality which have not met the standard. Based on the early research 
outcomes, methods of teachers in delivering instructional materials especially the material of Islam development 
history in the World which is still in less motivating ways for the students to be more active, creative, innovative, 
and have a joyful instructional. To create such an atmosphere used inkuiri-transactional approach. 

Instructional approach with inquiry-transactional is an instructional process that combines method of 
inquiry and market strategy. Method of inquiry has the key feature that emphasizes learning activities in the process 
of thinking critically and analytically students to seek and find their own answers to the question of a problem. 
Transactional method has the key feature of the adopted transaction information transaction between buyers and 
sellers of goods on the market. In the market there are strategic of transaction of information activities. Transaction 
information is intended as trading conditions in the atmosphere such as market information. Information is the 
subject of the transaction is a matter of Islamic education in the field of History of Islam in the World. 
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Chart 1 shows as transaction of information activities 
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Methods 

This study was made to develop instructional materials of Islamic Education in History which is 
appropriate for the students of Vocational High School. Referring to the objectives of the study then it is necessary 
to determine the appropriate method. 

This study uses research and development that combines two models, namely, first, Gall model. Gall, & 
Borg eighth edition (of 2007) which adopted the Systematic Design of Instructional by Dick, Carey, & Carey and 
second, Borg & Gall model fourth edition (of 1979). The step of early research of this study uses concept of research 
and collection of information from the model of Borg & Gall (of 1979) included the “research and information 
collecting - includes review of literature, classroom observations, and preparation of report of state of the art”. The 
initial research includes gathering of information-including literature review, classroom observation, and the current 
state. 

Research and development method that uses this initial research are considered appropriate to be used in 
this study for two reasons. First, using initial research which has been useful to get three important information, 
namely (a) the implementation of instructional over the years, including the use of PAISMK text book of grade XII 
2 nd semester in the field of history as well as identifying its weaknesses; (b) information on the expectation of 
stakeholders regarding instructional of PAISMK grade XII 2 nd semester in the field of ideal history, (c) The 
formulation of instructional objectives of PAISMK grade XII 2nd semester in the relevant field of history for 
vocational high school students. Second , literature study which is part of the initial research to help getting the 
theoretical concept of ideal instructional for PAISMK grade XII 2nd semester in the field of history. 

The steps of research and development of PAI instructional material model - in History for Vocational 
High School. 

Step 1 : Initial research. At this initial step conducted after the outcomes of proposal seminar and obtained 
research permit from Post-graduate program of State University of Jakarta in February 2013. The activities done at 
the initial research is to find the information or data regarding PAI instructional problems in the field of Islam 
development history in the world which is about the instructional materials, the sequence of instructional activities, 
the content outline, methods, media, time, and solutions on Islamic Education (PAI) instructional on the materials of 
history, namely history of the development of Islam on the materials of history, namely history of the development 
of Islam in the World in Vocational High School (SMK) for the next grade XII 2 nd semester. Richard A. Schwier & 
Farl R. Misanchuk said that “studying the audience will also provide some indications as to whether interactive 
multimedia is necessary or desirable”. From this sentence it can be understood that when looking for information 
from many sides such as students or teachers will be known the things that is really wanted in the instructional 
process, such as whether interactive multimedia is needed in schools. The data obtained through interviews, 
questionnaires, and observations in the environment of SMKN 1 Kotamadya Pagar Alam, namely from the 
principal, teachers, students, education department and the society. 

Step 2: Identifying the General Instructional Objectives. On the second step, after knowing the information 
and need on the initial research step, the next is to determine what is the general behavior (information) that are 


457 


expected to be mastered by the students after studying the historical development of Islam in the world. The 
formulation of the general instructional objectives obtained from various sources. First, from the curriculum of PAI 
which is a reflection of the government expectations as graduates user. Second , from the opinion of teachers, 
principals, and officials of Education Department of Pagar Alam as the education organizers. Third, from the 
opinion of the students of Vocational High School as the instructional subject actors. Based on the third party then it 
is formulated the general behavior expected to be achieved by the students after studying PAI of Islam development 
history in the world. This expected general behavior made as the general instructional objectives of PAI of Islam 
development history in the world for the Vocational High School. 

Step 3: Performing Instructional Analysis. On the third step, the next step is to outline the general behavior 
of into particular behavior and the connection of those particular behavior one and another. The connection shows 
the behavior structure. There is hierarchical structure, procedural, grouping or combination, in this step, it is 
obtained the behavior map or competence that show the sequence of behavior from the simplest or low to the 
complex or high behavior. 

Step 4: Analyzing the Students and Context. On the fourth step, the next step is to analyze the current 
students. Through interviews with the teachers identified two things. First, early behavior or early competency 
which has been mastered by the students before instructional PAI of Islam development history in the world. By 
knowing the early competence then it can determine the list of behavior which do not need to be taught and the 
behavior that needs to be taught. The limit of both lists is realized by dotted line called the input behavior line. The 
map of behavior with the dotted line is shown in Chapter IV. Second, the initial characteristic of Vocational High 
School which is related about the age of Vocational High School Students grade XII, information about the history 
of Islam development in addition to the Islam development history in the world they studied at grade X, XI, and XII 
in the odd semester, as well as the subjects they get in relation with the level of reasoning. At his step it can be 
obtained the information about the characteristic of Vocational High School Students grade XII 2 nd semester who 
study about the history of Islam development in the world and know about the diversity in understanding the history 
of Islam development in the world. 

Step 5: Writing the Objectives of Performance or Objectives of Specific Instructional. On the fifth step, the 
next step taken is to write the objectives of specific instructional about the competence expected to be mastered by 
the students after the instructional. The specific instructional objectives are the outcomes of instructional analysis. It 
contains four elements namely A element (audience) of the students, which in this case the student of Vocational 
High School, B element (behavior) the behavior expected to be mastered by the students after the end of the 
instructional, C element (Condition) the condition at the time behavior is tested and D element (degree) criteria for 
successful behavior. 

Step 6: Developing the Assessment Instrument. On the sixth step, the next step made is to develop the 
assessment that can measure behavior or competence of the student such as those set forth in the objectives of 
general and specific instructional. The development of assessment instrument prepared by way of making instalment 
plan and continued with preparing test items consistent with the competencies contained in the instructional 
objectives. Assessment instalment of instaictional outcomes developed consist of 10 test items of multiple choice to 
measure the comprehension of students (ability to explain) about the history of Islam development in the world. In 
addition, the assessment of student instaictional outcomes using various criteria namely attendance, creativity in 
making and displaying attributes, activeness at the time of information transaction, and the making of paper. 

Step 7: Developing Instructional Strategy. On the seventh step, the next step made is to identify the strategy 
used in the instructional to achieve the objectives. This strategy started by encouraging student instaictional as pre- 
instinctional activities. The activity stimulates the motivation and focus the attention of the students to the material 
discussed. Furthermore the instructional strategy followed by the presentation of the material, giving examples and 
demonstrations, participation and active student assessment, and follow-up of activities related to newly learned 
skills to be applied in the real world. This instructional strategy prepared using inquiry-transactional approach with 
the following steps: 

a. First step Preparation : Praying, Classifying the Group, Displaying test items, and Preparing Data 

b. Second step Initial Activities : Praying, Apperception, Presenting TIU and ICT, Watching Filmstrip, 
and Displaying test items 

c. Third step Core activity 1 : Setting the Stand 
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d. Fourth step Core activity 2 : information transaction 

e. Fifth step Core activity 3 : confirmation 

f. Sixth step Core activity 4 : evaluation 

g. Seventh step Final Activity : Closing 


Chart 2 shows 7 steps inkuiri-transactional process 



Step 8: Developing Instructional Materials. On the eighth step, the next step made is using the instructional 
strategy to have instructional materials. The instructional materials to be instructional physical models developed in 
the research in the form of: 

a. Textbook printed materials and CD, about the Islam Education for the Vocational High School grade 
XII 2 nd semester in the History of Islam Development in the World. 

b. Teachers Manual, on how to use the instructional materials in the implementation of instructional. 

c. Students Manual, which is about how the students learn printed materials like text book and CD. This 
student manual directly integrated or blends into the textbooks and not made separately as Teacher 
Manual. The integration is to facilitate and accelerate the students in instructional instructional 
materials and ensure that surely read by students. 

Chart 3 shows 8 steps of research and development 
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Result 


Step 1'. initial research result. The first step in the research is initial research made in September 2012 to 
February 2013. This activity is to find information and data about instructional needs, problems, and ideas on 
Islamic Education instructional solutions (PAI) in the material of history of the development of Islam in the world at 
the Vocational High School (SMK) grade XII 2 nd semester. Information and data obtained through interviews with 
various people in SMKN 1 Kotamadya Pagar Alam, namely the principal, 3 teachers of PAI, 3 graduate students, 2 
officials of Department of Education, 2 officials of the Ministry of Religious Affairs of city and province, Mayor of 
Pagar Alam, and the surrounding community (including the students’ parents). 

The information and data of the initial research includes the instructional on the development of Islam in 
the world which is used now and suggestions on its improvement for the future. This information covers all aspects 
of instructional. In detail, the information includes the material scope, currency, completeness, and vastness, the use 
of teacher manual, student manual, the availability of instructional textbooks and CD / VCD, simplicity to be 
understood, the obstacles encountered when teachers teach the material of history, sequence of instructional 
activities, outline of the existing contents / materials, the next instructional materials, measurements of student 
instructional outcomes, the most effective measurement, classroom situation, instructional methods, media and 
instructional instruments, and available instructional time. 

The outcomes of the initial research are as follows: 

(1) The text book used is the PAI textbook of SMK grade XII published by the Ministry of Religious 
Affairs, Yudistira, and Armico. The three textbooks used by the teachers in this school known that seen from its 
scope is adequate and complete because it covers 5 continents. Asia, Africa, America, Australia, and Europe. But 
one of the 3 textbooks used do not describe the scope in detail, but only briefly. According to the respondents, the 
material of history of the development of Islam in the world that is in the three of these textbooks is less advanced 
for it is not up to date. The explanation of material in the book just a glimpse of the five continents and the vastness 
considered to be less because only explain about some countries for each continent. 

(2) The instructional objective in the field of history for the students of SMK grade XII 2nd semester is 
referring to contents standard of 2006 which has the competence standard which contained to understand the history 
of Islam development in the world. While its basic competency is explaining, giving examples, and taking lessons of 
development of Islam in the world. But the expectations of the teachers that students can learn in detail about the 
development of Islam in the five continents and at the end of the lesson students can make a report or paper on 
history of the development of Islam in the world according to their understanding. 

(3) In the instructional process, the teacher uses guideline in the form of lesson plan and syllabus. In the 
textbooks used by teachers in teaching have not found teacher manual, student manual and other media, including 
CD /VCD. 

(4) Instructional instalments used by teachers just textbooks guidance, syllabi and lesson plans, markers 
and whiteboards. Expectations of teachers in the future that the instructional process can use instalments and media 
that can support the instiuctional process that is active, creative, and fun, for example by using the instalments and 
media such as movies, maps, globes, and pictures of Muslim leaders that packed in CD / VCD. 

(5) Method or model of instructional that is used by almost all teachers is lecture method and occasionally 
alternated with question and answer, discussion, and summarize the material assignment. According to the teachers, 
only by this method it is considered appropriate with the material because they lack of instalments and media, the 
students seem do not like the history material. This is proved because many students get sleepy during the 
instiuctional process and some others (about 20%) rarely go to class. According to one of the teachers, said that 
Mechanical Engineering in the school is less concerned with the material of Islam development history in the world. 
The expectations of teachers in the future there will be a method of instaictional of Islam development history in the 
world that make the students to be active, creative, and happy supported by the effective and efficient instalments 
and media. If it happens, then the students will certainly have a long memory of the history of the development of 
Islam in the world. 

According to some respondents recently that the teachers in Pagar Alam is not implementing the 
instiuctional models appropriate to the material yet. They still use the conventional method or the old model of 
teaching methods such as lectures. So far, most teachers just explain and the students just noted. The instructional 
process continues without any innovation. The teachers realize that they should find new innovation by using model 
of instructional that can improve student instiuctional outcomes, but they did not do that because they do not feel to 
responsible for that. For the promotion of teachers it is required to make scientific work in the form of CAR 
(Classroom Action Research), so inevitably the teacher should make this scientific work. 
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Department of Education expects by the presence of new models of instructional so that the students can be 
active and teachers can also be active. Some suggestions that arose in relation to the model or method of teachers in 
teaching is that the material of history of the development of Islam can be presented using some teaching strategies 
such as holding fragment, role play, the students play practical history so that easier to understand and interesting, 
then uncover and tell the facts of history through fdms about the history of the development of Islam in different 
continents, so that the students learn not only about worship but also to know the differences so that not only can 
blame others. 

(6) The media used by the teachers should be in the form of slide by displaying map with the red or yellow 
dots to show the location of states with the Islam development, displaying historical fdms, globe, pictures, etc. The 
research result of : Professionalism Portrait of Yogyakarta Teachers in Teaching and Instructional Activities” 
conducted by the Education Research Network of Yogyakarta (JP2KY) in early 2010 showed that 75 percents 
teachers of the research participant have not use the instructional media in teaching. 

(7) Time. The instructional of history of Islam development in the world listed in the curriculum of 2006 
was 6 hours lessons or 3 times meeting. But often do not have time because the material of history of Islam 
development in the world is in the last meeting and the teachers are running out of time. Uninteresting instructional 
process make absorptive capacity of students in the subject is not optimal. The students expected that this material 
has a instructional process that can make them active, not sleepy, can listen to the songs and watch movie, interact 
with friends, learning in group, and feels fun. 

Step 2: Identifying the general instructional objectives. The general instructional objectives in the 
curriculum of 2006 used the competence standard term. Competency standards for the material of history of the 
development of Islam in the world for Vocational High School are to understand the development of Islam in the 
world. Particular instructional objectives in the curriculum of 2006 used the term basic competence. This research 
identified objectives of instructional about the historical development of Islam in the world for PAI subject of SMK 
grade XII 2 nd semester which can be regarded terminal objectives of historical development of Islamic instructional 
in the world are: 

“ the students are able to make a report or papers on historical development of Islam in the world”. 

The instructional objectives are more appropriately called the terminal objectives of historical development of 
Islamic instructional in the world, because the general instructional objectives used for PAI as a whole. 

Step 3: Instructional analysis result. The third step, this research and development is to determine step by 
step competence that should be achieved by the students to achieve the terminal objectives by looking at the sub- 
skills needed for the final objectives mastery of the historical development of Islam in the world. The special skills 
are cognitive skills or knowledge, and attitudes needed by the students to achieve the terminal objectives. The 
instructional analyzes objectives shown in the following chart: 
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Chart 4 shows a result of instructional analysis 



Step 4\ Analysis outcomes of students and contexts. The fourth step in this research and development is 
analyzing the students in parallel, context in which they will learn the skills, and the context in which they will use 
it. Current students' skills, preferences, and attitudes are determined along with the characteristics of instructional 
settings and the settings in which skills will eventually be used. At this step the information obtained from the initial 
research outcomes on the characteristic of students of Vocational High School grade XII 2 nd semester who learn 
about the history of the development of Islam in the world. 

The characteristics of students of Pagar Alam Vocational High School grade XII who will study about the 
history of the development of Islam in the world are: 

(1) All students of grade XII 100% Muslim (data attached). The students will be better able to absorb the 
Islam values when the instructional process of history of Islam development in the world than if there is non-Muslim 
student. This characteristic provides an opportunity for the teachers to be better to instill the Islamic values through 
inquiry-transactional instructional approach. 

(2) Aged about 15-17 years. At this age the students of Vocational High School already has the ability to 
develop the reasoning in the study of historical development of Islam in the world. Vocational High School students 
who are over 15 years old already has enough maturity in thinking and reasoning so they categorized able to study 
the history of development of and values of Islam. 

(3) Have studied and learned about the historical development of Islam during Bani Umayyad and Bani 
Abbasid in PAI subject of grade X, Middle Ages and modern age in PAI subject of grade XI, and historical 
development of Islam in Indonesia on PAI subject grade XII 1st semester. They also have studied several subjects 
that support reasoning creativity such as mathematics and physics that allows students to absorb the abstract material 
such as history which aims instilling values of Islam through the instructional analysis of historical development of 
Islamic in different continents. 

(4) Having diverse competencies namely smart, moderate, and not smart. These characteristics suggest the 
need of instructional that involves discussion and group work as well as instructional from the students in the other 
groups, like become one of the main features of inquiry-transactional approach. 

(5) In general have been able to operate the computer and can access the internet. It is produced from the a 
practical subject in the computer laboratory, while for internet access the students still using the internet access via 
mobile phones and internet cafe because at the school there is no access to the internet. Based on the characteristics 
in this research the instructional material used are taken from various sources such as text books, general books 
about the historical development of Islam, the internet (via mobile phone and internet cafe), newspapers, and 
magazines. 
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(6) In general having a tendency to get bored quickly and sleepy when following lessons with a boring 
presentation. Therefore, by this research it is presented with the inquiry-transactional approach, not dominated by 
the lecture method. Thus instructional materials become more enjoyable. 

Step 5: Performance objective formulation outcomes. The fifth step in the research and development is to 
write performance objectives or specific instructional objectives in the curriculum of current PAI called basic 
competence. The performance objective contains competence which is expected to be mastered by the students 
when they finish the instructional. The specific competence identified in the analysis process of terminal objectives, 
identifying the skills to be learned, the conditions in which the skills will be demonstrated, and the criteria for 
successful performance. At this step will be formulated the specific instructional objectives on historical 
development of Islam in the world are that students are expected to be able to: 

Based on some specific instructional objectives above and analysis of instructional then formulated the 
specific instructional objectives in the material of historical development of Islam in the world are that students are 
expected to be able to: 

(1) Compare the development of Islam in five continents. 

(2) Show examples of the development of Islam in the world (figures, relics, various fields of science, 
etc.). 

(3) Take the values from the historical development of Islam in the world. 

Step 6: Instillment development outcomes of instructional outcomes assessment. On the sixth step of this 
research and development is to develop the assessment instrument that can measure the students’ competence as 
desired in the objectives. At this step, the research continued on the development of assessment instrument designed 
to measure the students’ achievements in the competency in the instructional objective of PAI of Vocational High 
School grade XII 2 nd semester which is making report or paper on the historical development of Islam in the world. 
The table of blueprint of process assessment instrument and instructional outcomes looks as follows: 

Table Ishow elements of instructional process assessment 


No 

Elements to be valued 

Quality 

Skills 

Individual 

Group 

1 

Attendance 

5 % 

V 

V 

2 

Creativity in making and 
displaying attributes 

20% 

V 

V 

3 

Their activeness at the time of 
information transaction 

10% 

V 

V 

4 

Multiple Choice Test 

15% 

V 

V 

5 

Paper 

50% 

V 

V 


Total 

100% 




From the table shown in the attachment it can be interpreted that the instructional outcome value of each 
student is very depending on the success of group instructional process and less depending on the individual 
instructional process. This is one of the characteristic of instructional based on inquiry-transactional approach. The 
instructional approach emphasizes on the role of collaboration in the instructional process and the results are joint 
result. So, the instructional outcomes value in the inquiry-transactional emphasizes on the group instructional 
process. 

Before making list of question or instructional outcomes test, the blueprint was made first. Richard I. 
Arends explains that “the test blueprint is a devise invented by evaluations specialist to help make these decisions 
and to determine how much space to allocate to certain kinds of knowledge and to the different levels of student 
cognitive processes”. 

Step 7: Instructional strategy development outcomes. On the seventh step in this research and development 
is to identify the strategy to be used in the instructional to reach the objectives. This strategy will emphasizes the 
component to encourage the students instructional including pre-instmctional activities such as stimulates the 
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motivation and focus of the attention, the presentation of new material using inquiry-Transactional and the 
assessment of students activeness, and follow-up of the activities related to newly learned skills to be applied in the 
real world. 

Table 2 show of instructional strategy 


Steps of Instruction Activity 

Content 

Method 

Media 

Time 

1 

2 

3 

4 

5 

Step of Introduction 

First step Preparation : Praying, 
Classifying the Group, 
Displaying test items, and 
Preparing Date 





Step of Presentation 

Second step Initial Activities : 
Praying^ Apperception, 
PresentingTIU end ICT 
Watching Filmstrip, and 
Displaying test items 





Third step Core activity 1 : 
Setting the Stand 





Fourth step Core activity 2 : 
information transaction 





Fifth step Core activity 3 : 
confirmation 





Sixth step Core activity 4 : 
evaluation 





Step of Dosing 

Seventh step Final Activity : 
discuss the test, the teacher 
teaching, games, and 
closing 





Total of Time 



Step 8: Instructional materials development result. The eighth step in this research and development is to 
use the instructional strategy and instrument of process and instructional outcomes assessment to have instructional 
materials draft. This instructional materials draft includes the instructional material (textbooks and CD) which the 
students manual, teachers manual, and instructional outcomes assessment instrument are integrated in it. 

Conclusion 

The development of instructional materials based on inquiry-transactional includes 8 steps, namely initial 
research, identification of objectives, instructional analysis, analysis of students and context, writing performance 
objectives, developing assessment instruments, developing instructional strategy, and developing instructional 
materials. The result is in the form of one set of instructional materials draft, namely lof textbook for the students, 1 
of teachers manual, and 1 of interactive CD. The draft of the instructional materials still need to be evaluated in the 
next research efforts. 
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Abstract 

Nurse practitioner students are educated to compose Subjective, Objective, Assessment, Plan (SOAP) 
notes, a method for consistent communication of patient case information. However, since student clinical 
experiences cannot be controlled, each student participates in a different variety of clinical patient conditions. This 
qualitative study evaluates the usability, functionality, and perceived value of ‘‘Advanced Practice”, a free and open 
source software mobile application designed to assist with peer-to-peer sharing of SOAP notes. Four nurse 
practitioner students were given access to the mobile application and interviewed within a focus group. Student 
responses suggest that the application is usable, works as expected and perceived valuable as a learning support or 
reference tool for professional examinations. Additional SOAP notes that could be recommended to support 
instructional content, content organization that fits the patient cases versus the nursing disciplines, and a 
commenting system were the primary recommendations. 
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Introduction 


Nurse practitioner students are educated to compose clinical case studies in the organizational form of 
“SOAP” notes, an acronym for Subjective data, Objective data, Assessment, Plan. In these case studies, which are 
developed by the students throughout their program of instruction, students provide an analysis of their patient’s 
problem, a rationale for their working diagnosis, and reasoning behind the patient care plan. However, since student 
clinical experiences cannot be controlled, each student participates in a different variety of clinical patient 
conditions. Motivating students to review peer SOAP notes is a way to increase the student’s exposure to a variety 
of clinical conditions. Thus, a mobile application, “Advanced Practice” to support and promote a nurse practitioner 
student’s peer review and collaboration, was designed and developed. More specifically, the purpose of developing 
“Advanced Practice” was to 1) to create an infrastructure for students to securely share SOAP notes, 2) to increase 
engagement and critical thinking by adding simple interactivity and 3) to create tools that facilitate discussion and 
review of SOAP notes. 

Advanced Practice is intended to afford students the opportunity to be both consumers and producers of 
information. This critical skill for the digital world is also a part of practice within the medical community. The 
experience of sharing case studies with peers and conducting meaningful discourse around clinical events is a 
practice typically reserved for graduates already employed in the professional healthcare provider field. Advanced 
Practice gives our students the opportunity to practice this experience and generate meaningful learning events for 
themselves and their peers. 

We have conducted research to evaluate the mobile application Advanced Practice, in terms of usability of 
the interface, functionality, and users’ perceived value. Thus, this paper introduces the application and presents the 
preliminary findings of our research. 


Literature Review 

The Advanced Practice app is a new technology designed to support an educational environment amongst 
the nurse practitioner students, an approach of growing adoption within the educational technology field (Definition, 
2008). Specifically, the application allows for the sharing of domain knowledge between nursing students. This 
exchange of experiences is a primary component for communities of practice (Wenger, McDermott, & Snyder), and 
access through a shared portal is critical to the success of electronic communities of practice specifically (Ho et al., 
2010 ). 

SOAP notes entered into Advanced Practice are organized in brief, self-contained, reusable, aggregated, 
and metadata tagged blocks known as Reusable Learning Objects (RLOs) (Beck 2010). This content structure has 
been seen as particularly effective with health science students (Childs, Blenkinsopp, Hall & Walton, 2005; 

Wharrad, Kent, Allcock & Wood, 200 1 ) and is well adapted for mobile devices. However, to ensure the tools 
efficacy towards supporting a student-centric community of practice, it’s necessary to ensure accuracy, data entered 
by the students, application usability, and student and patient privacy (Demiris, 2006)(Richardson & Cooper, 2003). 

Advanced Practice is designed using the Integrative Learning Design (ILD) framework. 1LD is a broadly 
applicable cross-disciplinary framework that includes not only pedagogical approaches to instructional design and 
research, but also the integration of those elements into iterative technological development processes (Bannan- 
Ritland, 2003). ILD’s roots in instructional design, usage-centered approaches, and distribution to audiences beyond 
the target audience make for an appropriate fit towards development of a mobile learning tool within nursing. 
Advanced Practice and its subsequent design are highly supported by the combination of instructors and students 
identifying gaps and working with instructional designers to propose solutions. Usage-centered approaches and 
feedback loop processes focus on usability, identified by human computing interaction studies as a necessity for 
smartphone and tablet design. Finally, the program design is modular as to facilitate transfer to other academic fields 
interested in case study based applications. 


Design 

The application is structured with a typical linear progression, which is consistent with the SOAP case 
structure. Upon loading for the first time, the user must register for a username and password before they can login to 
the application (Figure 1). This authentication process and the secure storage of the cases are handled by the 
Appcelerator platform that the application is developed in. This security layer allows the application administrators to 
control who is allowed access to the SOAP database and which SOAP cases the user will be able to see. This 
functionality is intended to support student privacy while facilitating future research through content delivery control. 
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/ 



Android 

Figure 1. The structure of the application 


Once logged in, the student is presented with four categories of SOAP cases. These were selected by the 
team to reflect the primary educational tracts taken by students in the department. Below the categories is a list of 
case studies that the student can select and each case has a short 50-character description and case number (Figure 
2). The short description is intended to inform the user of the SOAP case topic and for sharing case study 
recommendations by peers. 

Welcome, Jonathan Log out 



Advanced Practice 


Adults | Geriatrics I & Pediatrics I ^ Women 



28 year old male with vomiting and diarrhea 


Figure 2. Screenshot of a case in the Adult category. 

The research team decided to maintain a primarily text implementation of SOAP notes rather than 
incorporate multimedia elements for the following reasons: 

• The addition of multimedia would have been a modification from the original assignment adding more 
work to an already tight curriculum. The researchers hope to explore and assess the potential educational 
value of this addition in future revisions. 
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• Mobile learning for nursing is still a young field and early implementation research still holds great value to 
the community’s body of knowledge. 

• Patient privacy issues as they relate to multimedia need to be explored in greater depth and appropriate 
educational content delivered to ensure students are given proper guidelines. 


When a SOAP note is selected, the user experience follows the traditional linear sections of subjective data, 
objective data, assessment and plan (Figure 3). One minor addition was made to the SOAP structure in order to 
create critical thinking moments during the case review. The application enables the user to decide the differential 
diagnosis or plan they would implement prior to seeing the author’s recommendation. Though the correct choice 
may be clear to the student writing the SOAP, this effort requires the student to consider plausible alternatives as 
well as point out misleading facts. 



Subjective 


Objective 

Vital signs are as follows; Ht. 70 cm, Wt. 9 
kg , (50 be a well-nourished and well- 
developed infant. She is alert and active, 
fussy but consolable by mother. Physical 
examination is unremarkable, HEENT. 
cardiopulmonary and abdominal 
examination is benign, A complete blood 
count (CBC) with differential, urine analysis 
(UA), blood culture and sensitivity, and urine 
culture and sensitivity are ordered STAT and 
available for review. CBC with differential: 
Hgb: 9.5g/dl, WBC- 10,000, 50% neutrophils, 
46% lymphocytes, 3% monocytes, 3% 
eosinophils. The UA reveals cloudy urine, 
leukocyte esterase and nitrite. The blood 



□ 
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Select the most appropriate diagnosis 
|~1 Urinary Tract Infection (Cystitis) 
f~1 Systemic Viral Infection 
|~1 Pyelonephritis 


Submit 




Acetaminophen 90mg PO q4-6hours PRN 
for fever and fussiness. Do not exceed 5 
doses in 24 hours 


□ Orapred 9mg PO qid to decrease 
inflammatory process. 


Acetaminophen 90mg PO q4-6hours PRN 
for fever and fussiness. Do not exceed 5 
doses in 24 hours 


□ 


k. A 


Figure 3. Linear approach to displaying subjective, objective, assessment and plan. 


Instead of listing the choices within a paragraph as eliminated options, the previous static text 
implementation, the application creates discrete opportunities for critical thinking and selection. The learner receives 
immediate feedback on the appropriateness of their decision and is able to move forward to the next part of the case 
study (Figure 4). 


Assestment 

Case: 28 year old male with vomiting and diarrhea 

O Viral gastroenteritis 
O Giardiasis 

® Bacterial gastroenteritis 


Submit 



Assestment 

Case: 28 year old male with vomiting and diarrhea 


Viral gastroenteritis 

Correct. Viruses are responsible for 80% of gastroenteritis in the U.S. Classic 
symptoms include nausea, vomiting, non-bkxxJy diarrhea, abdominal pain, and 
low-grade fever. 

Giardiasis 


Incorrect. Recent travel to an endemic country makes the case suspicious for 
protozoal infection however the patient’s symptoms do not quite match. 
Symptoms of giardiasis include frequent, sulfuric-smelling belching, diarrhea, 
and abdominal bloating/cramping after meals. 

Bacterial gastroenteritis 


Incorrect. Bacterial gastroenteritis is usually associated with high fever, or 
bloody/severe diarrhea. 



Figure 4. Users receive immediate feedback in Advanced Practice 
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Technology 

The Advanced Practice application is being developed as Free and Open Source Software (FOSS) due to its 
benefits for an unfunded educational technology project. FOSS development facilitates the applications adoption by 
reducing ownership issues and allows the project to potentially benefit from peer-to-peer development with other 
interested institutions. Appcelerator is used as the development platform due to its active development community 
and because it allows for cross-platform distribution including iOS, Android, and Web. 

Context 

Advanced Practice was designed in parallel with a class of nurse practitioners being introduced to the 
format and value of SOAP notes. Students who had previously taken the class and nursing faculty were the primary 
informers of the original design with current students as participants in evaluating the mobile application. Advanced 
Practice is being distributed by installing directly to participant devices or through the web interface. We anticipate 
submitting the first version to platform specific application stores by the end of 2013. 

Research Design and Participants 

The Advanced Practice application described above was implemented as a research tool in this study. At 
the beginning of fall 2013 semester, the application was introduced to nursing students, who were enrolled in a 
graduate level course. In total, 12 students were registered in the course. For the study, students were asked to 
voluntarily participate in using the application as part of their required practice working with SOAP notes. Students 
were not offered any incentives to participate except access to an early version of the mobile application. Four out of 
12 students in the course volunteered. One student was female and three were male. 

The participants accessed the application by either downloading it to their own mobile device (cell phone 
or tablet) or by accessing the mobile friendly web version. To measure Advanced Practice’s interface usability, 
functionality, and users’ perceived value, a survey and a 40-minute focus group was conducted. 

Results 


Interface usability 

During the focus group, participants were asked about the pros and cons of using the application for 
reviewing SOAP notes. Students’ responses were generally positive stating that the application was convenient, easy 
to use and user-friendly. When asked about organization of the SOAPs, students were perplexed by the current 
categories used for classification. They suggested it would be helpful to have categories more refined, based on for 
example chief complaint instead of degree track. 

Functionality 

Participants were asked to discuss how they felt about maintaining access or expanding viewership access 
to the content. Students showed mixed opinions. Students recognized both the benefits and the challenges of 
expanding viewership. Students were receptive to the idea of expanded viewership as there would be more cases of 
greater breadth and depth. Students weren’t concerned if people outside of their class authored cases as long as the 
content was correct. 

Some discussion developed around the value of complimenting textbook material with SOAP notes. One 
student cautioned that an emphasis should be kept on evidence based cases “instead of claiming something is right” 
such as in a textbook example. Students suggested a place for comments at the end of a SOAP for continued 
discussion around the topic. 

User perceived value 

When asked about utilizing this application within their coursework, students agreed that the tool did 
facilitate their learning. In particular, students recommended having an instructor identify SOAP notes relevant to 
textbook material to enhance course content, but that reviewing notes should not be graded. Generally, respondents 
suggested that requiring a grade for an activity within the application might deter students’ motivations towards the 
application as a learning tool. Overall students saw the app as greater facilitation to learning outside the classroom 
compared to current methods. 

Students recognized the value in the application as a learning tool, but didn’t think it would be as useful 
once they graduated and entered the profession of nursing. However, with improvements to the application and more 
cases to select from, some students thought the application would be useful to review in preparation for their board 
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examinations. One student could foresee authoring and submitting a case based on their professional experience 
once in the workplace, especially if the case was unusual in nature. 

Additional comments 

At the end of the focus group, students offered some final comments on the application. Patient privacy was 
a concern that students wanted to see insured. Multimedia incorporated into a case was also thought to be beneficial 
for learning, e.g. the sound of an irregular heartbeat embedded in a case. 

Discussion and Conclusion 

Advanced Practice provides a contextualized learning experience that allows students to generate 
meaningful learning events to themselves and their peers. Nursing students often receive limited exposure to real 
world rare or unusual patient cases while learning textbook material. SOAP notes provide an avenue for 
complementary exposure through a contextualized learning experience. To realize this goal, the results indicate that 
we have further work and research to do. 

Students would like to see categories refined beyond adult, geriatrics, pediatric, and women’s health. 
Organizing by chief complaint would allow students to use the application as more of a reference tool. It appears 
that there is a clear need to provide more meaning to SOAP notes outside of the actual material but within the 
category. This contextual information provides an impetus for a student to select and begin on-demand learning. 

One solution may be a feature in the application that allows the instructor to create and share a collection of SOAP 
notes pertinent to a topic being covered in class. Sub categories may be an additional solution toward adding greater 
value around a case. For example, a clinical student making a patient experience, at a later time, could look up the 
chief complaint to reinforce the topic with similar cases. 

A place to leave comments on a case was also suggested by students. Having a dialogue around a case may 
further reinforce community of practice behavior. However, if the app is to foster conversation around each SOAP 
note then comments, which tend to be individually centered and fragmented (Lu & Churchill, 2012), may not be the 
best solution. Further research is required to test the potential value of a comment system and explore how the 
instructor could utilize this information within the class environment. 

The connotation of grading deterred students’ attitudes towards approaching and using the application. 

Since students author content at one point, the rigor of the case may diminish as a student realized that their choices 
within a case would ultimately be evaluated and therefore creating lower level cases may be of a greater benefit to 
them. Currently, it is not in the design plan to grade students on the options they choose. However, the instructor 
would like to know who and how many cases are being accessed. Would this deter from a community of practice as 
the learning motivation moves towards being centered on points? This is a question to explore in future research. 
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Abstract 

This article addresses how to approach different pedagogies depending on the orientation of blended instruction. 
Four blended courses with different orientations will be presented in this article. The article will categorize the 
model of each orientation of blended instruction and different approach to pedagogy and course design. 

Introduction 

The paper illustrates some major shifts in twenty-first century higher education through the exploration of a 
blended course which went through a redesigning process to achieve greater success under the umbrella of newly 
emerging educational milieus, where higher education increasingly offers more flexible, online, blended modes of 
instruction for time-strained students to be able to have access to learning. Blended learning, one of the most 
adoptable modes of instruction, helped as well as challenged educators despite its reputation as “the single-greatest 
unrecognized trend in higher education today,” as the president of Pennsylvania State University said (Young, 

2002). Ever since the online course management system has been available in the field of education, blended 
learning has been a naturally developing organism evolving according to educational purposes and contexts. 

Blended learning has moved arbitrarily between traditional and online instruction which resulted in ambiguity of 
what, how, and when educators mix different delivery methods of instruction. 

Definition of Blended Learning 

Blended learning is a formal education program in which a student learns at least in part through online 
delivery of content and instruction with some element of student control over time, pace, and/or space, and at least 
in part at a supervised brick-and-mortar location away from home. Additionally, blended learning is called different 
terms such as distributed learning, open and flexible learning, and hybrid learning. The arbitrary nature of blended 
learning reverberates in the different definitions of the term: 

• Combining instructional modalities (or delivery media) (Bersin & Associates, 2003; Orey, 2002a, 
2002b; Singh & Reed, 2001; Thomson, 2002) 

• Combining instructional methods (Driscoll, 2002; House, 2002; Rosset, 2002) 

• Combining online and face-to-face instruction (Reay, 2001; Rooney, 2003; Sands, 2002; Ward & 
LaBranche, 2003; Young, 2002) 

Much interest in blended learning has been exhibited by researchers, but the critical discussion of 
understanding and the reconceptualization of teaching from the instructor’s point of view have been largely absent 
from the blended learning literature. Most research has been confined to the definition, direction, and the broad 
goals that educators might espouse as they design blended environments. 

The Future of Hybrid Model 

The global revolution in information technology, which has transformed the international economy, is also 
destined to transform American education (Moe, Cuban, & Chubb, 2009). Hybrid schools facilitate the 
incorporation of a wide selection of educational technological innovations that transform the education process, 
which is why Moe and Chubb predict that most schools of the future will take a hybrid form (Moe and Chubb, 

2009). 
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Driving the movement toward hybridization is recognition of the considerable benefits that technology 
offers to the learning process. The promise of technology extends beyond customization of the curriculum and mode 
of learning. This “force of liberation” (Moe and Chubb, 2009) frees schools from geographic constraints, improves 
accountability by way of sophisticated data management systems that evaluate progress continuously, and provides 
significant savings in labor, which can make funds available for use in more effective ways, as the case studies have 
demonstrated. Furthermore, in the educational sector, computer-based technologies hold the capacity to remediate, 
accelerate, review, preview, supplement or supplant the existing teaching/learning system. Networking capabilities 
mean resources can be borrowed from or shared with other educators and students around the country or globe, 
expanding access to the top-tier teachers and instructional materials (Hassel and Hassel, 2009). 

No one best technology exists; instead, the suite of tools evolves daily. As more schools enter the digital 
terrain, the incentive for technology firms to invest in improvements of their systems and/or content expands. 
Inevitably, in such a system, quality will rise and prices will fall. Hybridization of traditional pedagogical models 
presents a unique opportunity for schools to fight low enrollment and boost efficiency. By substituting specialized 
software for expensive college-trained workers for a portion of the school day, schools can significantly cut costs 
and reinvest those savings in more productive ways (Jacob, 2011). 

A hybrid school model combines both online, computer-based learning with traditional classroom learning. 
While still attending a “brick-and-mortar” school structure, students in hybrid schools spend all or some of the day 
taking online classes or utilizing instructional software under the supervision of school staff. The delivery of 
education is thus a hybrid of the online and traditional models and can be customized to the individual student. 

Smart assessments allow students to skip content they are familiar with and progress to more challenging levels. 
Data collection systems provide reports to teachers regarding which lesson plans can be crafted to address areas of 
high need or interest. The pace of instruction can be modified on an individual basis, and the format for assignments 
contains a broad scope from crafting traditional essays to creating screen casts, podcasts, or video blogs. The 
development of hybrid schools should be unsurprising, considering the impact of technological innovation on 
modern society over the past 40 years (Jacob, 2011). 

The Orientation of the Blended Learning 

The choices of blended learning entirely change pedagogical approaches to blended instruction and the 
components of course design. This article addresses how to approach the different pedagogies depending on the 
orientation of blended instruction. Four different blended courses with different orientations will be discussed. There 
are three different kinds of blended courses according to the definitions of the University System of Georgia (USG): 
(1) A partially-at-a-distance course uses technology to deliver more than 50 percent of class sessions, but visits to a 
classroom are required. If a course is offered through two-way interactive video, then it should be coded partially at 
a distance because students must meet at a designated location. (2) A hybrid course uses technology to deliver 50 
percent or less of class sessions, but at least one class session is replaced by technology. (3) A technology-enhanced 
class uses technology in delivering instruction to all students in the section, but no class sessions are replaced by 
technology. 

The types of blended learning defined by USG are based on the proportion of use of technology in a class 
session. Contrary to USG’s definitions, this article discusses the different types of blended instruction defined by 
their orientations other than the proportion of use of online instruction or technology. The orientation of the blended 
instruction is critical because it decides the entire approach to the instruction in terms of design and pedagogy of 
instruction. 

Four Models of Blended Learning 

The following are definitions of the models that are based on the orientation of blended learning as well as 
the related case studies that differentiate the pedagogical approaches to blended learning. 

1. Learning Enhancement Model: Case Study One (Kim, 2013a) 

Learning Enhancement Model is a model in which, within a given course or subject, instructors choose to 
facilitate online learning components to enhance students’ learning experiences in the course. The major cause of 
this model is to compensate for the limitation of face-to-face course components. In the following section, the article 
illustrates the case study of blended learning based on a learning enhancement model (Kim, 2013a). 
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a. Start of Blended Learning 

Difficulties associated with student teaching relate to time and scheduling constraints, which limit 
opportunities for student teachers to communicate reactions, reflections, and questions raised after entering into their 
internship. This has resulted in the lack of opportunity for student teachers to process and make sense of their 
experiences at field sites by applying and connecting concepts from coursework and by sharing their questions and 
reflections with peers. The interchange of questions and reflections about learners, pedagogy, and teachers' 
instructional and classroom management behavior, between and among student teachers, is essential for making 
sense of the complexity of the social macrocosm of classrooms and schools. Thus, specific ways to foster such 
interchange among student teachers was explored and described. 

b. Pedagogical Structure of a Blended Course 

The pedagogy for this specific blended course for future teachers was based on the solid theory and 
incorporated into the course design, such as problem-solving approach advocated by Bransford and Stein (1993). 
Gagne (1980) said, “The central point of education is to teach people to think, to use their rational powers, to 
become better problem solvers.” Since learning to teach is far from a simple process, and is predominantly 
associated with ill-structured problems, teacher educators need to seek a way to help student teachers build the 
ability to locate, understand, and respond to the dense and multi-faceted problems of the classroom. As an effort to 
build student teachers’ problem-solving and reflection skills, this study facilitated the problem-solving approach, 
particularly the IDEAL approach, advocated by Bransford and Stein (1993). The IDEAL approach was used to help 
student teachers frame the way of approaching problems in the classroom. The IDEAL approach consists of the 
following steps: 1) Identify problems and opportunities; 2) Define goals; 3) Explore possible strategies; 4) 

Anticipate outcomes and act; 5) Look back and learn. The student teachers will locate their problematic cases and 
look for the strategies consistent with their goals in literature and, if needed, seek professional assistance from 
professors and cooperating teachers. To extend and build student teachers’ problem-solving and reflection skills 
through socially constructed interaction, student teachers were given access to and invited to participate in Web- 
based discussion through an online classroom management system called VISTA. The VISTA discussion board was 
a space for student teachers to share the problems, solutions, and implementations of their practice during internship. 
The university supervisor posted the weekly discussion topics based on the steps of the IDEAL approach; student 
teachers were required to comment on peer student teachers’ postings and respond to other comments. 

c. Improving the Quality of Interaction and Problem-Solving Skills 

This study assessed the quality of interaction and of learning experiences of student teachers during 
computer conferencing while learning problem-solving skills. To examine the quality of interaction, this study 
examined student teachers’ discussion contents by using the Interaction Analysis Model (IAM). The IAM analyzes 
the online discussion by using five phases: 1) sharing and comparing information; 2) discovery and exploration of 
dissonance or inconsistency or advanced teaching strategies; 3) negotiation of meaning/co-construction of 
knowledge; 4) testing and modification of proposed strategies or co-construction; 5) metacognitive 
statements/applications of newly constructed meaning. This study offers implications on whether use of computer- 
conferencing in student teaching is effective enough to increase student teachers’ problem-solving skills and 
knowledge construction as well as to prove the fact that computer conferencing is an instrumental medium for 
quality discussion among student teachers. This case study is the first stage of research that investigates the nature of 
the dialogic processes generated among student teachers in an online discussion group. 

The interview and descriptive survey results showed the positive responses of all student teachers toward 
use of online forums during student teaching. One of the significant findings was that student teachers appreciated 
interaction opportunities with peer students. An opportunity to communicate with other student teachers was not in 
the curriculum of their internship program. In addition, student teachers were often scattered and isolated from peer 
student teachers even in the same building. As a result, the internship experiences were limited to interaction 
between the intern and the cooperating teacher in the classroom. 

Student teachers valued the interaction with people at the same level as them, non-hierarchical interactions 
absent in interaction with cooperating teachers or university supervisors. It seemed conversing with other student 
teachers created a comfort zone where there was no tension, which was inevitably generated by the evaluation 
authority of the cooperating teacher or university supervisor. 

These student teachers enjoyed getting different perspectives on the same issue and seeing how their peers 
dealt with the same problem. The peers’ different perspective gave student teachers a new way of looking at a 
solution. The collaborative discussion created a new scheme: that peers can be a valuable resource from whom they 
could learn. This phenomenon was closely related to Vygotsky (1978)’s proposition of social constructivism. The 
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collaborative discussion in computer-mediated communication (CMC) further created emotional support. Student 
teachers often felt vulnerable in their new role. Worries over their execution in the classroom generated tension and 
stress during the internship period. The evolving stress was released to a great extent after they gained knowledge 
that other interns faced similar issues in their class. It was a pleasant relief for student teachers to discover that 
others faced either the same or very similar issues. 

Another significant advantage to interacting with peers in online forums was that this design of online 
forum allowed student teachers to step back and reflect on the problem. They then could consider their goal for this 
issue and broke down the problem-solving approach into steps. One of the most significant findings was the fact that 
they learned to approach a problem in a more systematic way. 

In this study, we showcased examples of interaction analysis for examining social construction of 
knowledge. This study found higher mental functions of social construction of knowledge appeared throughout the 
postings in analyzing the progress of the entire discussion transcript. This study showed that the structure of the 
discussion contributed to higher mental functions in the collaborative discussion. The discussion format of this study 
guided students to follow the systematic steps to approach the problems: the IDEAL approach. Coincidently, there is 
an overlap between elements of the IAM model and those of the IDEAL approach. 

We believed that the algorithmic, collaborative discussion through CMC helped student teachers equipped 
with a frame of how to grapple with the problem. Frames impact how we see and make sense of our lives; frames 
help us to establish boundaries, name problems, form opinions, and uncover solutions (Entman, 1993; Goffman, 
1974; Judge, 1992). Schon explains how teachers frame challenging situations that emerge in their practice through 
“naming the problem, setting boundaries of attention to it, and imposing coherence to provide directions for change” 
(cited in Achinstein & Barret, 2004; p. 719). Framed within a context of shared knowledge with discussion 
participants, each student teacher learned from other practitioners to solve problems by experimenting and working 
toward a viable solution in their classroom. Frames can assist student teachers in assessing their mental archives for 
similar experiences and can help them adjust their practice accordingly (Schon, 1987). 

It is difficult to help individuals learn the ways of thinking and acting required by a profession. Learning to 
teach for this kind of practice is far from formulaic (Darling-Hammond, 2006; p. 40). It is up to teacher education to 
provide teachers in training with coursework and experiences to build on, challenge, and move beyond their 
perspectives and interpretations in order to see teaching with a wider lens (Kennedy, 1999; Wideen, Mayer-Smith, & 
Moon, 1998). We believed that incorporating CMC in student teaching with theory-grounded structure, such as the 
IDEAL approach, puts forward the needed training, which allowed student teacher to learn ways of thinking and 
acting requisite to the teaching profession. Through collaborative discussion practice, student teachers seemed to 
expand their ability to locate, understand, and respond to the dense and multi-faceted problems of the classroom. 
Markel (2001) asserted that the advantage of online discussion lies in allowing students time for reflection. Student 
teachers commented on positive effects of stepping back and breaking down the problems following the steps of the 
IDEAL approach, a process which was absent in their reflection process in the past experience. 

Another key effect of CMC in student teaching was to build a community of learners. A student-to-student 
communication scheme of online discussions — synchronous or asynchronous — played the major role as a tool to 
develop a learning community (Choi, 1999; Park & Kim, 2000). Participants of CMC discerned the value of 
collaborative online forums from which they received pedagogical, managerial, and emotional support. Participants 
ascertained viable, experimental teaching ideas from other participants and new classroom management approaches. 
Additionally, they were relieved from the stress encountering a new role in their life by sharing and seeing in 
practice similar or the identical issues experienced by others. They gained some consolation by seeing that their 
peers were not always successful. 

This study proved that computer conferencing is an instrumental medium for quality discussion among 
student teachers. Further, it demonstrated criticality of building a systematic design of the online forum in an effort 
to increase the higher mental function of online communication. In the process of the analysis of transcripts, we 
learned that there has been an absence of student-student interaction in student teaching, which could develop a 
different communication environment from interaction with cooperating teachers and university supervisors. 

The study of Amdiraal, Lcokhorst, Wubbels, Korthagen and Veen (1998) found that while CMC provided 
emotional support, it was not as effective for fostering reflection. In contrast, this study found that, a well-structured 
forum grounded in properly aligned theory could induce quality reflection. The significance of this study is the 
development of a systematic online forum that builds social construction of knowledge and promotes problem- 
solving skills for student teachers. The chief advantages of the online problem-solving forum model are: 

1 . Its appropriateness for using constructivist, collaborative student teaching learning contexts 

2. Its focus on research-based problem-solving skills 

3. Its integration between the IDEAL approach and IAM to check the quality of online forums 
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4. Its straightforwardness and simplicity of use 

5. Its adaptation of CMC in student teaching 

2. Accessibility Model: Case Study 2 (Kim, 2013b) 

According to the body of research on the topic of blended learning, there are three major rationales for 
blended learning: (1) Improved teaching and learning, (2) increased flexibility in and access to learning; and (3) cost 
effectiveness (Graham, 2006). The second kind of blended course was built based on the institutional needs of 
increasing the size of enrollment by increasing accessibility of the course. The course that the author had to teach 
was announced as a hybrid course and the bi-weekly face-to-face times were scheduled in the university course 
registration Web page. The rationale for using blended learning for this specific course was not a voluntary decision 
made by the instructor to improve teaching and learning outcomes. Blending the course with face-to-face and online 
instruction was an involuntary institutional requirement aimed at increasing flexibility in and access to learning. By 
its very nature, blended learning offers more flexibility to learners by reducing face-to-face time and by adding more 
online learning components. These features are especially attractive to mature learners who have to balance jobs and 
family responsibilities with their studying as well students who live at a distance. In the midst of the negative 
influence of economic turmoil, the institutional decision of increasing flexibility was necessary. Along with other 
higher education institutions, this medium-size, four-year state university has been challenged by a variety of 
roadblocks such as escalating costs, decreased funding, increased oversight and regulation, entrenched practices, 
outdated models, constraining policies, and low graduation and retention rates. In the midst of these chaotic 
challenges, the major concern of the university leadership has been increasing enrollment and retention. 

The face-to-face course the author has taught for many years became a blended-learning course which 
many researchers posited as “a transformative learning experiences (Garrison & Kanuka, 2004)” for students. The 
outcomes of this innovative modality of instruction were far from meaningful dialogue with peers, thoughtful 
reflection online, and increased student engagement (Ziegler, Paulus, & Woodside, 2006). By contrast, my blended 
course did not achieve the positive results of blended learning. Both parties, instructor and students, finished the 
semester with frustration and resentful feelings about the format of the course. The questions are raised: What have I 
done? What went wrong with the course? The author had to analyze the factors of unsatisfactory results in the 
process in implementing the blended course. 

a. Traditional Attitude: Lecture Obsession 

The lack of familiarity with technology tools in computer-mediated systems (CMS), which appeared to be 
one of the major reasons in the failed integration of two modalities of teaching, was not applied to the author’s case 
since she had majored in instructional technology for her doctorate. The author held enough technology skills to 
hold technology workshops for faculty members in her department. The major cause of the failed course was to 
manage class time throughout the semester. A blended approach permitted the instructor to change the way she uses 
the class time. The major concern came from lack of face-to-face time to deliver the course content in a given time. 
Sixteen weeks to deliver the course content was shrunk to eight weeks. All content seemed too important to 
eliminate any of the content. She has taught the same course successfully with high ratings of instruction 
effectiveness in her teaching evaluations. The component of her face-to-face course consisted of mainly lectures and 
class discussion along with students’ assignments. Because her presentation skills were highly favored by students 
(documented by her teaching evaluations), she was resistant to change the way that the course was taught. Because 
of the large volume of the content the author perceived as too important to eliminate, the instructor rushed to deliver 
the content in the reduced class time. Rushed face-to-face instruction led to students’ misunderstanding of the 
content and confusion at the end of classroom session. Because of the time constraint, no classroom discussions 
were held during the semester. Students’ feedback notes showed that lectures did not result in successful learning 
outcomes. Feedback she received from students also clearly indicated that too much information was presented 
during the face-to-face periods. 

The author had successful experiences in facilitating online components in her practice in the past. She 
used a discussion board as a space for student teachers to share the problems, solutions, and implementations of their 
practice during internship. As a university supervisor, she posted the weekly discussion topics based on the steps of 
the IDEAL approach advocated by Bransford and Stein (1993). Each week, student teachers were required to 
comment on peer student teachers’ postings and respond to other comments. This study proved that computer- 
conferencing was a beneficial instrumental medium for quality discussion among student teachers. This study also 
suggested that teacher educators should consider using online forums as training to help student teachers build on, 
challenge, and expand their perspectives and interpretations in order to improve problem-solving skills. Further, it 
demonstrated the criticality of building a systematic design of the online forum in an effort to increase the higher 
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mental functions of online communication. She also used online components of a discussion board to improve 
student learning. The online discussion board played a role as a checker to ascertain if students completed the 
reading assignments according to instruction given in the face-to-face class. The facilitation of blended learning 
seemed to open profitable educational opportunities in the previous blended-learning experiences. In contrast, this 
particular course seemed not to match the previous positive experiences. This course denied the effectiveness of the 
blended-learning course. After a long reflective process, the author came to the conclusion that there was only one 
factor that was not present in the previous blended courses: the reduction of the major component of the course, 
lecture. All other blended courses incorporated the online components of the course as a supplemental component to 
the original course curriculum. 

What was failing this particular blended course was the failure to transfer the lecture to comparable online 
components and to understand the strengths of both modalities of instruction. The next question will be how to 
approach this issue. Here are the pedagogical tasks to analyze the research and trials and errors of the future 
practice: 

1. Analyze the components of the course. 

2. Find the strengths of face-to-face instruction and online learning. 

3. Find the weaknesses of face-to-face instruction and online learning. 

4. Find the comparable component for the lecture, (screencaptured lectures and Voice-over PowerPoint slides) 

5. Find the weaknesses of comparable components and supplement additional components for the weaknesses. 

6. Extract the essence of the course content to reduce the volume of the lecture. 

b. Failure to Locate Compatible Online Components 

The lecture component of the class needed to find different modality to supplement the loss of the content 
of the lecture. Two major components were readily available as online components for blended learning: discussion 
board and quizzes. The quiz was facilitated as a tool to check students’ completion of required readings. These 
quizzes will ensure students’ completion of reviewing the screen-captured online lectures. Online discussion board 
was used to check the reading assignment, to encourage students’ reflection on learning, and to replace group 
collaboration. The author learned that online discussion is the tool to change the overall quality of the courses. 
Ways to facilitate the online discussion board are unlimited: from the plain talk of the topic to group discussion to 
sharing the technology-infused project. It is critical to know the possibilities to convert the discussion board space 
for the simple monologue-type discussion to a space for multiple educational activities which will enhance the 
collaboration and interactivity of the course. 

The online discussion board needs to be designed as a space where student knowledge is individually 
constructed and socially co-constmcted by learners. This approach is in accordance with the constructivist view of 
learning, which advocates student learning occurs in the milieu of their interpretations of experiences in 
constructivist learning environments (CLEs), rather than merely being transferred from the instructor (objectivist 
learning). The fundamental difference between CLEs and objectivist instruction is that students learn domain 
content in order to solve the problem, rather than solving the problem as an application of learning (Jonassen, 1998). 
The problem-solving skill, self-direction for learning, and collaboration are three major areas in 21 st century skills 
(The Partnership for 21st Century Skills, 2013) to be a globally competitive future worker. However, there are few 
course design models that address this significant issue. To address how to design a learning environment that 
produces the knowledge generator, rather than the knowledge consumer, this study experimented with a 
pedagogical model of constructivist learning environments (CLEs) that engage learners in meaning making 
(knowledge construction) (Jonassen, 1998). 

• Provide students with the authentic context of the learning task with the knowledge construction process 
(e.g., development of scenarios that help pre-service teachers to write lesson plans grounded in Piaget’s 
constructivist theory). 

• Have students use multiple modes of representation (e.g., creation of video clips to present Erikson’s 
psychosocial development). 

• Provide collaborative learning environments (e.g., facilitate online publishing websites (Wiki, Twitter, 
blogs, Ning and the like) to accumulate their collaborative learning in Brenfenbrenner’s bioeco logical 
model). 

c. Strengths and Weaknesses in Face-to-Face and Online Instruction 

For the best allocation of the particular components of face-to-face and online modalities, it is key to 
understand the strengths and weaknesses of both modalities. Strengths of both modalities need to be capitalized on 
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while the weaknesses of both modalities needs to be eschewed. Oftentimes, the strengths of online learning seem to 
be the weaknesses of its face-to-face counterpart. 

3. Instructor Discretion Model: Case Study 3 

The third orientation blended course was to increase the effectiveness of instruction where a math class was 
designed to increase the passing rate of a math class by incorporating the cutting edge technology — such as 
Camstudio and LivePen — which was intended to demonstrate the step-by-step process of math problem. There are 
two primary components of instruction. The first is traditional instruction, which delivers instruction to students on a 
face-to-face basis. The second is multimedia presentations, which incorporate audio, video, and interactivity using 
on-screen quizzes, cursor-manipulated tools, and navigation. Students will work at their own pace through a lesson 
strand correlated to the course objective. 

In a general sense, a math class is perceived as a difficult subject for an online course. Low math scores of 
students have been a major concern because of low student retention and graduation rates at many universities. 
Despite the general preconception of math as a difficult subject to convert to an online instruction mode, the 
instructor of a math course found the online component of the class to be useful for review for the difficult math 
process to follow. An online course’s accessibility (anywhere/anytime) allows students to view the recorded lectures 
repeatedly. Especially, thanks to the capacity of online instruction to allow students to adjust pace and time, the 
instructor found the online component beneficial for pre-review and review of complex math problems that students 
had struggled to solve. As a result, the blended mode of instruction improved teaching effectiveness and decreased 
the student failure rate. 

Recognizing the positive effects of computer-aided instruction, which has been scientifically proven to 
close achievement gaps for at-risk students (Barrow, Markman & Rouse, 2009), this study discusses an innovative 
way of teaching mathematics through integrating instructional computer tools into the traditional instruction by 
using such tools as WebAssign (an online instructional tool) and Bamboo (a graphics tablet). The Bamboo tablet is a 
computer-input device that enables a user to hand-write, similar to the way a teacher writes with a chalk on the 
blackboard. This is one way of showing how these technology tools can perform the Vygotsky constructs of 
scaffolding by analyzing the relative effectiveness of different hinting styles (scaffolding), learning styles, and 
cognitive development levels. 

The study found that how students with different learning styles benefit from distinct hinting formats from 
different modalities of instruction (such as only texts, audio instruction, and video instruction with math experts). 
Vygotsky(1978)’s term "scaffolding" is used to describe tutorial interventions or decisions serving this research 
project. "Hints" are problem-solving assistance that gives additional support in ways that improve the chances that 
the learner will be able to solve a problem. Another form of help is “problem-decomposition scaffolding.” In this 
method of problem decomposition, the math problem is broken into components, which allows individuals to solve a 
problem more easily. The hinting session, one of the components of instruction, will offer clarity that would not be 
possible without the use of technology by providing multimodal, interactive, gradual presentations that convey the 
concept effectively. Pedagogically enhanced math instruction empowered by multimedia capabilities enriches the 
lesson presentations, self-pacing learning, and instantaneous feedback. 

4. Cost-effective Model: Case Study (Kim, 2010) 

The fourth kind of blended course was developed due to the strain on financial resources of a professional 
development initiative for in-service teachers. The reduction of cost was the major concern in the course 
development of this kind of instruction because either the training organization or trainees could not afford time and 
money to attend educational sessions. 

The technology grant project was launched to help a county offset reduced state funding of technology 
training and close the achievement gap between economically disadvantaged students and non-economically 
disadvantaged students by providing technology resources to all schools. The staff development plan was designed 
as part of a grant project to improve teachers’ instruction through the implementation of technology and research- 
based instructional practices as well as to develop departmental curriculum and design program evaluations. 

Economic turmoil led the grant project team to consider innovative ideas to overcome the budgetary 
limitations through implementation of the grant. A limited time frame and funding were major considerations when 
the training team developed the “Professional Learning Sequence.” Three assumptions were made: 1) All training 
does not have to be face-to-face training. 2) Teachers will need additional onsite support after training. 3) Grant 
schools should agree to the new paradigm of training. According to the new training plan, 50 hours of professional 
learning is placed in three different formats: 1) face-to-face; 2) collaborative team session; 3) digital online 
instruction. As follow-up to the training, supplemental onsite support was provided. The face-to-face training 
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focuses were: 1) direct instruction, which includes lectures, demonstrations, and hand-on activities; 2) collaborative 
sharing and redelivery, which encourages learning as a community; 3) lesson planning, which involves the 
construction of instructional activities using technology resources. Two hours of collaborative team sessions were 
conducted during the planning period and at the end of training sessions. Two teachers per team shared the group 
ideas. Digital online instruction sessions were conducted for 20 hours. Course content sites were created. These 
included educational technology reading assignments, reflection assignments using blogs and wikis along with a list 
of sites and articles, online projects, and lesson preparation and planning assignments. 

This study investigated the extent of the perceptions of teachers on the effectiveness of the constructed 
hybrid technology training sessions to effectively meet their needs. If technology is to be used by students and their 
learning outcomes, well-structured technology training is mandatory. Since budgetary constraints limit providing 
on-site training, the traditional method of offering professional development, a critical need exists for exploring the 
effectiveness of fully online or hybrid training with online alternatives that will lead teachers to possess the 
confidence, understanding, and skills to effectively incorporate technology into their teaching practices. 

Pedagogical Approaches to Professional Development 

Increased access to information through new technologies, along with the need to prepare children to 
compete in an emerging information-based global economy, promises to fundamentally reshape school practice as 
we move into the next century (Harvey & Purnell, 1995; Jonasson, 1993). Despite increased access to computers 
and related technology for students and teachers, schools are experiencing difficulty in effectively integrating these 
technologies into existing curricula. 

According to the U.S. Congress, Office of Technology Assessment (1995), the lack of teacher training is 
one of the greatest roadblocks to integrating technology into a school’s curriculum. That same report revealed that 
most school districts spend less than 15 percent of their technology budgets on teacher training and development. 

Thus, Moursund’s (1992) contention that school systems have not taken into account the training and 
support teachers needed to appropriately and effectively use computer-related technology in the classroom remains 
relevant. The following elements are the pedagogical considerations that the professional developers and school 
administrators should take into account in planning professional development initiatives. 

Time: Teachers must have substantial time if they are going to acquire and, in turn, transfer to the 
classroom the knowledge and skills necessary to effectively and completely infuse technology into their curricular 
areas (Hawkins & MacMillan, 1993). However, Harvey and Purnell (1995) suggest there is an overwhelming 
sentiment that schools have yet to create the kind of training and practice time teachers need in order to learn how to 
effectively integrate technology into the curriculum. 

Taking into account varying needs: When designing staff development sessions on technology, 
individual differences must be addressed and individual strengths supplemented (Shelton & Jones, 1996). 

Sustained staff development. To help teachers properly complete the "learning cycle" of computer-related 
professional development, training must be ongoing and systematic (Kinnaman, 1990). 

Flexibility of professional development opportunities: Staff training programs designed for the 
technological development of teachers are effective when programming offers flexibility and is not based on a "one 
size fits all" philosophy. Teacher training programs must not expect that all participants will leave with the 
knowledge and skills to facilitate the transfer of learning to their individual classrooms. Instead, Browne and Ritchie 
(1991), Harvey and Purnell (1995), and Stager (1995) state that effective staff development for technology requires 
flexible content and opportunities. 

Provisional support: One of the most effective ways to align staff development with district/school goals 
is to invest in someone with experience in both technology and curriculum (Kinnaman, 1990). 

Collaborative development: The environment in which the effective technological development of 
teachers occurs is built around collaborative learning. Because teachers vary in their level of expertise at the time of 
their training, the context which surrounds their technological professional development must provide a non- 
threatening environment that is sensitive to the individual teacher's level of expertise and experience (Browne & 
Ritchie, 1991; Shelton & Jones, 1996). As a result. Stager (1995), Browne and Ritchie (1991) suggests that 
collaborative problem solving and cooperative learning must undergird the approach to technology learning for 
teachers. 

Linking technology and educational objectives: The technological training must have an instructional 
focus that guides teachers to think first about their curriculum which, in turn, helps them address how to integrate 
technology into the curriculum (Guhlin, 1996; Persky, 1990). 

Intellectual and professional stimulation: The model of staff development for technology must put the 
teacher/learner at the center of the learning experience and provide a meaningful context for learning (Stager, 1995). 
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Teachers need instruction that engages them and forces them to reflect on the benefits and limitations of teaching 
with technology (Persky, 1990; Shelton & Jones, 1996). When teachers engage with others in ongoing reflection 
about what they have learned about the instructional use of technology, they are more likely to critically evaluate 
their own pedagogical practice and redesign their instruction. 

Conclusion 

Despite its high demand and the rapid adoption of the blended mode of instruction in higher education, the 
knowledge base of blended instruction is still in its infancy stage. There are myriad areas that call for better 
understanding before settling upon the definitive mix of the Web-based instruction and its face-to-face counterpart. 
An instructor of hybrid courses needs to have a deep understanding of how people learn, what students’ learning 
styles are, and what technology can provide for the successful design of technology-integrated learning 
environments (Bransford, Brown, & Cocking, 2002). 

It is important to note that many school operators have implemented more than one blended-learning model 
for their students. The types of blended learning have been typically defined on the basis of the proportion of use of 
technology in a class session. Contrary to the traditional method of categorizing the kinds of blended instruction, this 
article discusses the different types of blended instruction defined by their orientations other than the proportion of 
use of online instruction or technology. The author found that the orientation of blended instruction is critical 
because it decides the entire approach to the instruction in terms of design and pedagogy of instruction. 

This article discusses how the same instructor found opportunities as well as challenges at the same time in 
implementing blended instruction depending on the orientation of the blended learning. That means not all blended 
instruction is effective for all learning environments. According to the case studies presented in this article, the 
success of blended learning is dependent upon understanding the nature of the instruction, algorithmic preparation of 
instruction based on the analysis of the course, locating comparable online (technology) components to traditional 
counterparts of instruction, understanding strengths and weaknesses of different learning modalities, and 
incorporating pedagogically effective educational theories into the course design. This article highlights empirical 
insights for pedagogical approaches to different mixes of blended learning and practical strategies grounded in the 
practice of university faculty members. Ideally, this will lead to readers reflecting on their own instruction and 
considering ways to develop a successful blended course. 
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Abstract 

The purpose of the study is to develop an instructional design model to support learners' continuous 
reflection during the learning process in cultural diversity education. Current instructional models for cultural 
diversity education are inappropriately utilized in school settings due to the lack of specific guidelines for 
instructors’ and learners’ activities and their insufficient explanation and usability. To enhance cultural sensibility 
and value internalization for cultural diversity education, learners’ continuous reflection during the learning process 
should be integrated in the instructional design process. Using the design-based research methodology, this study 
first derived a draft of instructional design principles from multicultural education literature review and monitoring 
classes, then suggested an instructional design model with the cooperation from expert instructional designers and 
field experts (teachers), and finally developed a practical instructional model through gradual model elaboration 
process. The features of the developed instructional design model are as follows. First, it presents specific learning 
stages and activity elements for cultural diversity education. Second, it provides different types of reflection 
supporting tools that an instructor can use for each learning stage in learner-centered design as well as specific 
examples of usages. Third, the developed model considers not only the cognitive aspect of learners, but also the 
affective one, suggesting instructional design elements to consider in order to cultivate attitudes or values. 

Introduction 

Needs for the multicultural or cultural diversity education have been heightened as the Korean society 
becomes a multicultural society. The definition of cultural diversity refers to various methods in which a culture of a 
group or society is expressed (UNESCO, 2008) and it is an approach to minimize the structural discrimination and 
socio-economical separation based on ethnicity, class, gender, religion, disability (Lee, Joo, & Kim, 2013). Banks 
and Banks (2010) stated that the goal of the cultural diversity education is for all students to acquire knowledge, 
attitude, skills to communicate, negotiate meaning and interact with various groups of people. In order to guarantee 
the effectiveness of multicultural education, every students should be offered with continuous multicultural 
education (Martorella, & Beal, 2002; Parker, 2001; Savage, & Armstrong, 2000). 
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However, currently available programs intend to help students in multicultural family adapt in the Korean 
society. The limited scope of multicultural educational locus in current programs have been criticized for its focus 
on a group of minority to be adapted into a society (Cho et al, 2010; Hwang, & Yang, 2008; Park, 2011). In this 
context, interests in multicultural education which notes a parallel human relationship and power relationship 
between majority and minority have been increased recently in Korea(Kang, & Jang, 2009; Lee et al, 2013; Lim et 
al, 2012). 

On the other hand, it is rare to find an instructional design model that has detailed activity stages or guidelines 
that an instructional designer can refer to when designing cultural diversity education. The current lesson models for 
cultural diversity education have activities or stage names that are too abstract or general for teachers to follow 
through. Moreover, they are not appropriately utilized in school due to insufficient guidelines for instructor and 
students’ activities in each lesson stage and lack of explanation or usefulness (Eun, 2009; Kang, & Jang, 2009; Kim, 
& Kim, 2009). And, focus of cultural diversity education includes not only cognitive aspects of learning but also 
learners’ attitude and value formation. To improve learners’ attitude, value and interests, it has been suggested that 
organizing lessons around learners’ activities is useful (Heo, 2010; Kim, & Jin, 2004; Kwon, 2004; Lintner, 2005; 
Scott, 1999). However, activity-centered lesson does not guarantee to achieve the intended goals. Despite the fact 
that activity-centered learning is significant in several aspects, students just focus on doing specific activity itself, 
which results in different learning outcomes (Kim, & Jin, 2004). 

This is due to the absence of reflective learning, where a learner forms his or her consciousness and 
behavior through monitoring ones status and learning situation (Choi, & Lee, 2003; Duffy, Lowyck, Jonassen, & 
Welsh, 1993). Reflection has been proved to be a success factor in the learning process, but research on facilitating 
reflection has been done mainly in the web environment (Chung, 2007; Kim, & Kim 2002; Park, & Woo, 1999) and 
there has not been enough studies regarding strategies or tools that can support reflection in the face to face learning 
environment. 

Generally, the reflection activities are in the journal writing format implemented at the end of a series of 
lessons or in the form of reminiscing previous learning contents. Current methods require learner’s memory which 
may result in erroneous information, and it may not be less effective in terms of promptness. It is also ineffective in 
terms of value internalization method. Therefore, research guidelines for instruction design is needed to help 
learners to continuously reflect their learning process. 

The aim of this study is, therefore, to develop a continuous-reflection-centered instructional design models 
for cultural diversity education which enables learners to reflect continuously during the learning process in order to 
internalize the core value of culture diversity. 


Theoretical Backgrounds 

Multicultural Education and Cultural Diversity Education 

Recently, due to the influx of foreign workers and increase in international marriage, the Korean society is 
becoming a multicultural society by having ethnically and racially diverse people. Therefore, fostering members of 
society to respond sensitively in multicultural setting to recognize and respect differences has become an important 
issue in education. To fulfill an ideal society through cultural co-existence and positive interaction, the revised 
curriculum in 2007 reflected multicultural education (Hong, 2012). However, multicultural education currently 
implemented has been criticized for its focus in the assimilative educational contents which target minority groups 
(Hwang, & Yang, 2008; Kang, 2008). Current education offers only methods to integrate children from multicultural 
family or minority into the Korean society by offering training programs for Korean language and Korean cultural 
understanding as adaptive education (Koo, Park, & Seol, 2010). 

Due to the emergence of critiques, narrow scope of multiculturalism has been transcended and the necessity 
of cultural diversity education is all the more emphasized. The definition of cultural diversity refers to various 
methods in which a culture of a group or society is expressed (UNESCO, 2008), and it indicates the cultural 
difference between people in terms of language, clothes, the ways to form traditional society, concepts regarding 
ethics and religion and interaction. Cultural diversity awareness not only enables one to understand other cultural 
groups but also experience internal changes such as attitude or value to acquire open attitudes and flexible mindset 
toward other cultural groups. That is, cultural diversity education is to foster abilities to acquire sensitivity regarding 
different life styles, world views, and cultures and to participate in the conversation between different groups and 
countries (Banks & Banks, 2010). 
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Instructional Design Model for Cultural Diversity Education and Reflection 

The existing multicultural education programs have been mostly implemented by specifying multicultural 
education subjects in terms of curriculum content, subject matter, and teaching-learning methods or by teaching 
understanding of other cultures in social education. Teachers recognize the importance and necessity of multicultural 
education but have difficulties in its implementation because they’re not provided specific guidelines to conduct a 
lesson (Choi, & Mo, 2007; Park, Sung, & Cho, 2008). According to Cho et al.(20 10), teachers are not even certain 
that the school-provided multicultural education activities are valid, and they expect professional organizations to 
provide education programs and materials. 

For an effective lesson, it matters how the teacher makes use of the ample teaching-learning materials. This 
ensures the necessity of building an instructional design model for cultural diversity education. The following Table 
1 includes the current multicultural education lesson models 


Table 1 . The Existing Multicultural Education Lesson Model 


Preceding Research 

Multicultural Lesson Model Stages 

Kang, & Jang 
(2009) 

©Suggest Pluralism Task ©Investigate and Understand Pluralism Task ©Secure Plural 
Perspective ©Cooperatively Adjust Plural Perspective and Set Temporary Solution 
©Present and Evaluate ©Reflect on Decision Making Process 

Kim, & Kim 
(2009) 

©Suggest Problem of Conflict ©Recognize Multi-culture Coexistence ©Resolve Cultural 
Prejudice ©Practice Anti-prejudice Activity ©Internalize Value 

©Suggest Situation of Cultural Conflict ©Reverse Major Culture ©Unfamiliarize 
Culturally ©Think in the Minorities’ Shoes ©Seek Multi-culture Coexistence 

©Recognize Problem ©Add Minor Culture Related Content ©Recognize Relationship 
Between Major Culture and Minor Culture ©Settle and Develop 

Jho, & Park (2009) 

©Unfamiliarize Socio-cultural Myth ©Looking at World from Others’ Perspectives 
©Write and Present Inter-cultural Material ©Encourage and Seek Plan for Practice 

Eun (2009) 

©Admit Cultural Toadyism or Ethno Cultural Centrism ©Recognize Difference Between 
Own Culture and Other Culture ©Understand and Respect Own Culture and Other Culture 
©Reflect on Own Culture and Other Culture ©Handle Multicultural Problem and Conflict 

Park (2006) 

©Reveal Child’s Prejudice ©Understand Child Literature Content ©Correct 
Misunderstandings Leading to Prejudice ©Explore Prejudice Related Emotions ©Post 
Activity and Wrap Up 

Han (2006) 

©Watch Movies ©Check Movie Content ©Analyze Problems and Question Period 
©Debate 


These instruction models suggest different approaching methods depending on subject matter, goal, and 
learner level; or they suggest teaching-learning methods that utilize particular media. These methods are different 
but in general, they go through the process of recognizing, respecting and internalizing the cultural difference. 
However, the explanation of each stage’s activity does not provide specific guidelines that the teacher can refer to 
when he or she plans the lesson. Each stage should clearly state the teacher and learner’s roles and activities 

Also, in cultural diversity education, the lesson is organized focusing on the learner’s activity, and it leads 
the learners to understand concepts and internalize values through concrete experiences. The learner’s reflection, in 
this context, is proposed to be an important factor in the successful learning. When experiences are not connected to 
the reflection, learners tend to be absorbed in the activity or behavior itself, and this makes it difficult to achieve the 
intended goals (Kim, & Jin, 2004). Learner’s reflection is also included in the existing multicultural education 
instruction models, but the reflection process is just limited to the reflection journal writing at the end of each lesson 
period. The reflection done at the end of the lesson is less effective in terms of promptness such as reflecting in the 
middle of the lesson and being able to reflect the result of the reflection in the following behavior or activity. This 
means that it is needed that learners guided to reflect as the lesson persists (Chung, 2007). 

The importance of experiences and reflection through learners’ activities is emphasized in the Kolb(1984)’s 
experiential learning theory. Kolb(1984) defined experiential learning as a learning process that brings the learner’s 
behavior change and growth as they analytically observe and reflect on the concrete experiences and from these 
experiences, deducing abstract conceptualizing, i.e. principles that can be generalized regarding behaviors, and 
based on this, trying new behaviors and trying to experience new things with new points of view. In experiential 
learning, the learning process is cyclical and is progressed gradually. The learners’ active participation in concrete 
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experiences and their self-reflection towards the experiences are important in the learning process. Learning takes 
place in the process of integrating experience, concept, observation and behavior. Kolb’s experiential learning 
theory model is cyclical format of four stages: ‘concrete experience,’ ‘reflective observation,’ ‘abstract 
conceptualizing,’ and ‘active experiment.’ Kolb (1984)’s experiential learning theory may work as a theoretical 
frame of an instruction design model for cultural diversity that needs inter-cultural experience and constant 
reflection in that it emphasizes activities and reflections of learners. 

Reflection Supporting Tools 

Schon(1983, 1987) suggested two types of reflection. One is reflection in action which is about the process 
of learning, and the other is reflection on action which is about the result of learning. Learners monitor their thought 
and learning behavior by reflecting their learning process. This process is focusing on realistic tasks, asking oneself 
whether a series of activities to solve the tasks are conducted in a reasonable way and finding alternative solution 
and ruminating on the result of it (Dewey, 1933). Discussions on types of reflections imply that both process and 
result of learning should be reflected and through the whole process of learning, continuous reflection is needed. In 
addition, reflection is not a skill that is improved naturally but it is an acquirable skill through learning (Kim, 2012). 
When teachers provide learners with opportunities for reflection and encourage them and provide regular feedbacks 
to reinforce reflective learning, learners develop the ability to learn from their experience (Lowe, Rappolt, Jaglal, & 
Macdonald, 2007). Accordingly, research on methods hew in which the teachers support the learners’ reflection is 
important subject to study (Sawyer, 2006). 

The literatures on reflective thinking introduced the effect of various learning tools that could support 
reflective thinking. The typical tools for stimulating reflective thinking are reflection journal, concept map, 
reflection questions, and checklists (Song, 2009). Reflection journal asks learners to write a journal by organizing 
the questions and concerns which came up in their minds during learning. By writing a journal, the learners are not 
stopped at reflecting with their head but they are provided with opportunities to evaluate their learning experience 
deeply by stating their thoughts (Boud, 2001; Boud, Keogh, & Walker, 1985; Chung, 2003). Andmsyszyn and 
Davie(1997) reported the effectiveness of interactive journal writing in facilitating reflection in their study of 
reflection facilitation in computer mediated learning environment. 

Concept map was developed by Novak and Gowin(1984), and it means pictures using nodes and links to 
express a propositional statement. Concept map has structural figures and it shows concepts which are the 
relationship between ideas and knowledge in an organized manner (Jonassen, Grabowski, 1993). Accordingly, 
concept map as visual symbol helps learner recognize the information quick and easy way and see through the 
whole structure of related concepts beyond understanding a specific concept. Thus, concept map can be flexible 
classroom tool to assess the degree of understating. In addition. Kinchin and his colleagues (2000) reported after 
qualitatively analyzing the usefulness of concept map, it plays positive role to integrated newly learned concept with 
previous knowledge. 

Reflection questions are the easiest tool for teachers to utilize in their classroom. Students are able to 
ruminate on their learning objectives, process of learning, and the content of learning through reflection questions. 
Chung (2007) encouraged students to fill the worksheets and to use reflective questions time to time while 
performing activities in each step in problem solving task. In this research, worksheets of each step are proven to 
promote team process of problem solving thus helping students’ performing PBL activities. However, the use of 
reflective question was quite inactive because appropriate reflective questions suitable to the level of students were 
not offered. Therefore, the teachers should lead the classroom naturally and use appropriate questions considering 
the learning contents and the level of students (King, 1994). 

Checklist is a tool for students to monitor their process of learning (Burke, 2009). It is a way of making 
students record the frequencies of a specific behavior or check whenever this behavior appears after previously 
designating items they want to observe (Ahn, 1997). However, when students evaluate their process of learning in 
the classroom, the questions should be simple and easy to identify otherwise they might interfere with the flow of 
lecture (Conner, 1991). 

In this research, the specific principles to utilize reflection journal, concept map, checklist, and reflection 
questions as supporting tools for continuous reflection in multicultural education are suggested and improved. 
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Methods 


Design-Based Research 

Design-Based Research is about learning in the context which is conducted through recursive and 
systematic designing and research process on teaching strategies and tools(Design-Based Research Collective, 2003). 
Design-Based Research is usually dependent on the context of teaching and learning, and a great of part is done in 
the real classroom for improvement (Sloane & Gorard, 2003). Accordingly, in solving the complex problems of 
teaching environment, practical solutions can be suggested to be rooted in the theories (Barab & Squire, 2004; 

Brown, 1992). 

In this paper, researchers try to develop an optimal instructional design model for multicultural education 
though Design-Based Research. The suggested model can provide teachers with practical principles for designing in 
the context of classroom environment. Teachers could be provided with the information about the steps and stages to 
lead students to effectively acquire the values and the meaning of cultural diversity and how the learning activities in 
each steps be arranged. They can, also, get a great deal of ideas of supporting strategies for students to wholly 
understand of the values and the meaning of what teachers provide. Recursive implementation and evaluation made 
suggested model optimal. 

Research Procedure 

The initial model including steps and activities was developed through reviewing previously suggested 
multicultural education models. This initial model was revised through the feedback by the instructional design 
experts and the field teachers, and its applicability of revised model was verified through applying it to the real 
classroom environment. 

The development of the initial model 

The initial model was developed including steps and specific principles based on the literature review. 
Literature review was conducted in the area of present models for multicultural education(Banks, 2007; Eun, 2009; 
Han, 2006; Jho, & Park, 2009; Kang, & Jang, 2009; Kim, & Kim, 2009; Park, 2006) to construct the steps and 
stages of the optimal instructional model. An initial instructional design model was developed with the steps of 
introduction-development-closing. 

In addition, based on the experiential learning model of Kolb(1984), researchers classified the multicultural 
education classroom activities into four steps: concrete experience, reflective observation, abstract conceptualization, 
and active experiment and determined their purposes and necessity. In order to develop tools and strategies to lead 
and support learner’s continuous reflection during the class, related literature (Andmsyszyn & Davie, 1997; Boud, 
2001; Boud, Keogh, & Walker, 1985; Burke, 2009; Chung, 2003; Chung, 2007; Kinchin, Hay, & Adams, 2000; 

Kolb, 1984; Novak & Gowin, 1984; Song, 2009) were analyzed. Reflection journals, concept map, reflection 
questioning, and checklist were suggested as useful supporting tools for reflection by the literature review. 

The development of the second model 

Two instructional design experts who have experience as teachers and the one social studies expert who 
had taught at teachers' college reviewed the initial model suggested their opinions through individual face-to-face 
interview. Their opinions and suggestions were analyzed and developed into recommendations for improvement. 
Each expert had an interview with two researchers. The interviews were conducted focusing on validity based on the 
previous literatures and recommendations for improvement about the suggested model. The interviews were 
conducted in a semi-organized way, and one interviewer mainly asked questions and the other one instantly record 
and transcribed conversations with laptop. The contents of recorded conversations were classified into four types of 
information: theoretical background which the suggested model root in, opinions about the model in general, and the 
activities in each step. Recommendations for improvement were identified and coded. Considering the purposes of 
development and applicability, the experts’ suggestions were divided into the ones which can be applied and the 
others which cannot be after three times of discussion for decision making. As a result, the initial model was revised. 
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The development of the third model 

To increase the possibility of utilizing the model in schools, the model was tested for use by eight field 
teachers who have different years of teaching experiences. The field teachers were asked to especially reflect on the 
expected difficulties if they use the suggested model to plan their class. Three parts were especially stressed when 
the teachers reviewed the model. First, are the names of teach steps and activities appropriate? Second, are the 
activities of each step applicable? And are the application plan of reflection supporting tools suitable? Third, is it 
possible to design various multicultural education when following the suggested model? The teachers’ opinions 
were classified according to three questions. Just like the first revision, the suggestions were applied to the next 
version of model. The second model was refined based on the participants’ feedback and the final model was developed. 

The final model 

Real instructions were conducted by two teachers separately to see how the model is realized in the 
classroom. A researcher observed the instruction and conducted an interview with the teacher and the students. 
The teachers were mainly asked if there were any difficulties planning and using the reflection supporting tools 
and how the students responded. Four students had a group interview from each class, so eight students in total 
had interview and suggested their ideas. They were asked difficulties, advantages and any general ideas related to 
the reflection supporting tools they used in the classroom. The collected ideas and suggestions from the teachers 
and the students were considered when revising the explanations of reflection supporting tools and the final model 
was developed upon them. 


Results 


The Development of the Initial Model 


The instructional design model for continuous reflection in cultural diversity education was designed by 
considering cultural diversity education and reflective thinking. In this study, the types of activities were extracted 
by analyzing prior multicultural education model and its instructional goals. The initial instructional model like 
Figure 1 was developed, connecting the process with four major steps from Kolb’s theory of learning by experience. 




Apprehension of 
Cultural Situation 



(D Applying to other 
cultural situation 
(9) Internalization 
) Extending experience 


< Reflection Supporting 
Methods> 

- Reflection Questions 
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- Concept Map 

- Reflection Journal 


(f) Exploring cultural fact 
0 Finding out cultural 
differences and similarities 
(D Collecting cultural 
information 


Extension of 

Exploring of Cultural 

Cultural Experience 

Information 



Figure 1. The first instructional design model for continuous reflection in cultural diversity education 
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The effective reflection supporting methods for each step are like the follow. In the concrete experience 
step, it consists of two activities, ‘realizing cultural situation’ and ‘checking the task.’ The main process of this step 
is identifying the various cultural differences or situations in our lives and recognizing the learning contents for the 
class. The applicable reflection supporting methods in this step is ‘questioning’. It focuses on checking whether 
learners understand the main points of the learning contents and helps learners to recognize the learning goals. 

The reflective observation step includes a process of solving problems relating to the learning task from the 
previous step, concrete experience. First of all, learners find out the cultural facts in the situation, shown in the 
concrete experience step. From the cultural situation, they research how the differences appear in each situation, 
and study reasons of the similarities and differences among other cultures. Checklist and questioning is suggested 
to prevent learners from absorbing unconsciously into activities while forgetting its purpose. 

In the third step, abstract conceptualization, identify the cultural similarities and differences, and connect 
the cultural elements from the lesson logically to from concept of the culture. The concept map can be an effective 
supporting method in this step. It can be used for activities to help learners understand how the various concepts 
can be connected, showing the similarities and differences. 

In the last step of the model, learners extend the concept of culture by applying the conceptualized learning 
contents into different situation. From the process, they deviate from adopting fragmentary knowledge from the 
cultural situation, understand culture as life style of human kind, and internalize themselves to respect different 
style of living and expression mode. Furthermore, it makes learners apply those attitudes and perception to real life. 
To support completion of the learning process, students write a reflection journal about things they find out and 
want to know more from the class. It can be used to arrange the contents cognitively and affectively they learned. 
The Development of the Second Model 

The initial model was reviewed by expert for validity, availability, understandability, and universality of 
the model. Their opinions which are reflected in second model are like the followings. First, provide efficient 
explanation for the instructional model. Second, guidelines for the reflection supporting strategies and tools should 
be explained concretely. Even though the model is valid as an instructional model for cultural diversity education, it 
should contain more detailed guidelines for teachers so that they can use the model in the classes. Those suggestions 
throughout decision-making process among researchers applied into the second model like Figure 2. 

The form of the model and designation of each level is revised in second model. In the revised model, it 
separates subjects to learners and instructors and suggests major reflection supporting tool separately for each step. 
The names of each level from Kolb’s model of learning by experience are changed to the main activity in each step 
such as ‘recognizing cultural situation’, ‘exploring cultural information’, ‘conceptualizing cultural situation’, 
‘extending cultural experience.’ In addition, the last activity, extending experience, suggested in 10 th step is 
eliminated since it seems to be similar to the extending cultural experience for internalization of other culture. Also, 
to make the terms clear, we changed ‘question’ to ‘reflection question’ in reflection supporting tool. For the detailed 
guides, we revised ‘collecting cultural information’ to ‘finding reasons for cultural differences and similarities.’ 
Reflection supporting tools is revised only in checklist by modifying and supplementing questions and form of the 
checklist. The types of questions were changed from five statements to 0,X checking and elaborated the reflection 
questions considering each step, and eliminated repeated questions. 
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Figure 2. The second instructional design model 
The Development of the Third Model 

The second model was reviewed by 8 experts working in educational field: six of them majored in cultural 
diversity education and the rest has master’s degree in pedagogy. The field experts referred about the activities, 
instructors’ role, types of reflection supporting tools, and applications. Since activities suggested in model seem to 
be weighted to cognitive aspect, it needs to include more elements from affective, behavioral aspects. 

In addition, the opportunities to present about the learners’ feelings and thoughts after the activities should 
be contained in the steps. Majoring in cultural diversity education, some experts answered the model should support 
more effectively for general teachers. They suggested many guide materials such as various examples, checklist for 
instructor, etc. The necessity of revising terms to reveal the meanings of activities more clearly was also suggested. 

The opinions about checklist were the major issue in reflection supporting tool. The terms should be 
revised for students’ understandings and the contents should reflect aspects of cultural diversity education. Besides, 
using the checklist in the middle of the class should disturb the flow of the learning and it might be ineffective. 
There was a comment about the standardizing form of the checklist. While making concept map, the Venn diagram 
is suggested since the learners should recognize the similarities and differences of cultures during the activity. 

Considering those suggestions, we revised terms and content of the reflection supporting tools, added more 
detailed material explaining the model, maintaining the form of the second model. 

The Development of the Final Model 

The third revised model received usability test, applied in elementary school classes. We confirmed some 
suggestion to revise the model by interviewing teachers who planned and conducted class with the model and 
students from the classes. One of the teachers suggested the model needs to give plenty scope for teachers, which 
they can elastically use the model in planning classes. The other teacher referred learners should have some 
opportunities to learn about the process of making concept map and checklist which are reflection supporting tool. 

Most students mentioned that reflection supporting activities like questioning, concept map, checklist were 
helpful for their learning, since it guides what to do in learning process. Meanwhile, it is confirmed that even though 
reflection supporting tools can be applied to all steps, teachers assumed the tools as an absolute. From this result, we 
improved the model like Figure 3, combining reflection supporting tools together for all steps and filled the first 
letter from suggested steps in brackets. Moreover, the development of the model is completed suggesting the guide 
for teachers using the model by the context and educational environment and reflection supporting tools (Table 2, 
Table 3, Figure 4, Table 4) 
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Figure 3. The final instructional design model 
Table 2. Example of reflection questions 

Step 

Reflection Questions 

Apprehension of 
Cultural Situation 

• What is it about that I experienced (saw, heard)? 

• What is fact suggested in that I experienced (saw, heard)? 

• What is today’s lesson? 

Exploring of Cultural 
Information 

• What is the fact that I already know relating to the situation I experienced (saw, heard)? 

• What are the similarities between cultural facts I already know and the scene I experienced 
today? 

• What are the differences between cultural facts I already know and the scene I experienced 
today? 

• What is the reason of the difference between cultural facts I already know and the scene I 
experienced today? 

Conceptualization of 
Cultural Relation 

• What are the relations between collected information? 

• How can the collected information be united and organized? 

Extension of Cultural 
Experience 

• How can the learned facts apply to other cultural situation? 

• What kind of change will be brought in my cultural perspective and behavior after today’s class? 

• (in other cultural scenes) What will 1 do? What kind of happening is expected? 


Table 3. Example of reflection checklist 


Step 

Reflection Check-list Questions 

O/X 

Apprehension of 
Cultural Situation 

• Did I see, listen, react actively on the situation given in the class? 


• Do I know well about the situation I experienced now? 


Exploring of Cultural 
Information 

• Did I fully examine the situation to find the meanings of it? 


• Did I listen carefully to others observing situations around? 


• Am I collecting information relating to the learning task? 


• Is the collected information important and reliable? 


• Do I know what is additional information needed to understand the situation 
deeply? 
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Conceptualization of 
Cultural Relation 

• Do 1 know well about the learning subject? 


• Do I know what the key information is (to understand the situation)? 


• Do I know relations between learning subject and information I collected? 


Extension of 
Cultural Experience 

• Can I apply the content I learned today into my real life? 


• Can I relate the content I learned today to other situation? 




Table 4. Example of reflection journal 


Step 

Reflection Journal Questions 

Extension of Cultural 
Experience 

• What are the impressive things and memorable things among learning contents? 

• What did you feel during learning and after learning? 

• What are the experiences or examples nearby to which what you've learned could be applied? 

• What circumstance in the future might be related to what you've learned? 

• What are the difficulties and things which need further explanations? 


Discussion and Conclusions 

This research focused on methods to engage students in continuous reflection during learning process by 
developing instruction design model for cultural diversity education. The basis of this research is attributed from 
prior researches, which insisted that classes should be student centered to improve students’ interest and to form 
their attitudes or values (Heo, 2010; Kim, & Jin, 2004; Kwon, 2004; Lintner, 2005; Scott, 1999), and that the 
reflective self study is important in activity-centered classes (Choi, & Lee, 2003; Duffy, Lowyck, Jonassen, & 
Welsh, 1993). 

The discussion of developing instructional model for cultural diversity education is extended to three aspects 
as the followings. First, this research determined concrete learning steps and activities for cultural diversity education. 
The final instructional model is fulfilling the actual needs from education field by suggesting prescriptive form of 
concrete steps and activities for instructor and learners in cultural diversity education. Second, we suggested detailed 
examples of reflection supporting tools and of utilization in each step for student-centered classes. This research guided 
some reflection supporting tools for each step and method to provoke learners’ reflection in activity-centered class. 
Third, the specific instructional design elements to consider for affective domain such as learners’ attitude or value 
have been suggested along with instructional design model. Cultural diversity education is a cognitive, affective, 
behavioral education to make students acknowledge and respect value from different cultures, and live together. 
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The instructional design model contrived cognitive learning by step of apprehending cultural situation, 
exploring cultural information, and conceptualizing the situation. Besides, as experiencing diverse cultures, it made 
behavioral changes possible, by discovering reasons for cultural difference and similarity, and applying 
conceptualized cultural relations in other situation. Lastly, the instructional design model is suggested for changes in 
affective area relating to learners’ attitude and value by accepted cultural knowledge from internalization step. 

Meanwhile, some suggestions for effective application of the model, from findings in developing and 
applying process, are like the followings. First, to implement the cultural diversity education, instructors should use 
time-block or aware of other methods for flexible class management. Although the example of the class materials 
provided in this study is based on quantities of two classes (90 minutes), the actual classes was conducted in one class 
(40 minutes), considering the actual school environment. To ensure students play a leading role in exploring cultural 
information, conceptualizing its relation, and internalizing it, it takes a certain length of time. Thus, the key is to insure 
plenty time to induce learners’ deep thinking and reflection for internalizing value and changing attitudes in student- 
centered learning. Second, as using complicated reflection supporting tools would disturb the flow of class, it is 
important for instructors and learners to use simple and familiar reflection tool. To accomplish learning goals in the 
class without immersing oneself in activities, the reflection is suggested during class. However, the flow is rather 
distracted, when the time designed for reflection is too extended or complex tools are used in reflection process. 

Therefore, using simple and easy form of reflection tools to remind learning goals and meaning of activities 
is essential to maintain flow of the class. Third, various example should be suggested together for practical use of 
the model. Those who are not only familiar with concept of cultural diversity but also making effort to support 
continuous reflection need to spend much more time and effort to prepare. Indeed, some participants in this research 
who majored in multicultural education or are interested in cultural diversity education showed much faster 
applicability and understanding than ordinary teachers, who suffered understanding the model and preparing 
materials for classes. 
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Abstract 

Recently, inquiry-based learning has gained renewed emphasis, and has gradually become the central 
theme in educational research and practice, including the context of online learning. Meanwhile, innovative 
technologies provide new methods and enhancements of online inquiry-based learning. The purpose of this paper is 
to discuss key components of online learning and explore how innovative technologies, such as mobile apps, cloud 
computing and other technologies can be employed to enhance this type of learning. 

Introduction 

Online learning is currently expanding rapidly at universities, colleges, and even high schools. According 
to USNews, online course enrollment climbed for the 10 th straight year. Out of 6.7 million students, 32% were 
enrolled in online higher education in 2011, compared to 6.1 million in the previous year. In 2012, 62.4% of 
colleges offer fully online degree programs, compared to 32.5% ten years ago ( USnews , 2013). However, compared 
to the proliferation of offerings and options of online courses, there are the poor retention rates in online learning. 
The quality of the online program is one of the contributing reasons for learner dissatisfaction with this type of 
learning (Ruth, Sammons, & Poulin, 2007). 

Inquiry-based learning pedagogies can be used to solve this issue by increasing the quality of online 
learning, as well as learner satisfaction. Why? First, inquiry-based learning has been proven to have a very positive 
impact on learning from K-12 to college and graduate level education. The benefits students can gain from inquiry- 
based learning include: deep conceptual learning, higher-order thinking skills, and positive attitudes towards 
learning (Blumberg, 2000; Edelson, 2001; Zuckerman, Chudinova, & Khavkin, 1998). Second, inquiry-based 
learning has many different instructional models, such as the learning cycle (Edelson, 2001), and the authoring cycle 
(Short, 1996), among others, to meet different learning and teaching needs. Third, based on constructivist theoretical 
perspectives, inquiry-based learning has many elements that can help best produce learner satisfaction. Using 
authentic problem scenarios to engage students and collaborative groups to form a community for sharing and 
exchanging knowledge are two ways that enhance student motivation for learning through inquiry (Chinn & 
Malhotra, 2002; Hancock, Kaput, & Goldsmith, 1992). Finally, 21 st century learners are expected to have 
information, media and technology skills. Current inquiry-based learning pedagogy emphasizes that learning with 
technologies can help meet those goals. 

The focus on this paper is to explore possible strategies and tools that can be embedded into online course 
to bring the best practices of an inquiry classroom to an online environment. The suggestions in this paper are based 
on my dissertation on adult students’ online I-Search inquiry-based learning experiences. Therefore, I will situate the 
discussion for online adult learning at the college/university level. 
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Brief Description of My Dissertation Study 


In 2006-2007, 1 designed and conducted my dissertation study. My goal was to understand students’ online 
inquiry-based learning experiences. Five adult students participated in this study. They used an I-Search inquiry 
model for their project. I-Search is an inquiry-based model in which students are required to pursue a topic of their 
interest. They were asked to choose a topic that was interesting to them. Then they designed higher-order questions 
and researched relevant information from various resources, including the library, Internet, and experts in their field 
of interest. Finally, they wrote their I-Search paper to report their findings and reflect on their learning (Joyce & 
Tallman, 1997). The students have one time of face-to-face class at the beginning of the course, and the remaining 
course time was spent online. Each week, they had a 2-hour online class. They also used a WebCT forum to post 
their questions and make comments within their small group. Even though this I-Search model is mostly designed 
for teachers and librarians, this model has many of the essential elements emphasized by general inquiry-based 
learning. Therefore, the I-Search model can be applied to many other inquiry-based learning situations. Listed below 
are the components of inquiry-based learning matched with technologies and pedagogical strategies that demonstrate 
how online learning can support inquiry teaching methods 

Selecting Topic 

In inquiry-based learning, students are often allowed to select their own topics, as they did in my study. 
Students do not usually have a hard time finding their initial topics, but often they have difficulty narrowing it down 
enough for a course project. For example, one student’s topic was the history of the early Christian church. When 
she narrowed her topic down to researching the first believers, it was still a large topic for a course project. In this 
step, online instructors need to help students to realize that a topic can be approached from various perspectives. The 
topic should be narrowed down before the next step of inquiry learning. There are many tools that can be used to 
help during this step. For example, using online conceptual mapping tools or cloud-based conceptual mapping tools 
can not only help students brainstorm their potential topics, but also help them to visualize the possible ways to 
narrow down their topics, such as simply circling one part of their topic on a conceptual mapping tool (please see 
Table 1 for detail). 


Generating Questions 

Once a topic is selected and narrowed down, asking questions is the next step in the inquiry-based learning 
process (Short, 1996; White & Frederiksen, 1998). My study showed that many students had difficulty with this step 
and needed additional scaffolding for online learning. The assignment was for students to create higher-order 
questions that assist them in obtaining new knowledge, however, many students only created questions in which 
they were already familiar with the answers. It was challenging for them to determine what they do not know about 
their topic and therefore they had trouble determining appropriate questions. For example, a student is creating a 
travel plan to go to London. She has been there several times before and is knowledgeable about London. Although 
her questions may have fit the requirement of higher-order questions, she did not develop questions that provided 
her with new information. Another issue was that students can create inconsistent questions, in which secondary 
questions were created in a way that could not help answer the primary question. For example, a student wanted to 
know “How has the addition of man-made organizations taken us away from the model demonstrated by the 
community of the first believers (church)?” Her secondary questions were: (a) How is the church to be governed, 
according to Scripture? (b) How did the early church organize? (c) How do these compare? These secondary 
questions presented no way to help answer her primary research question. 

To facilitate generating questions online, teachers can also ask students to try to answer their own questions 
to discover their prior knowledge about the subject. It is also suggested to discuss questions through social networks 
or other communication software and to ask students to review questions of other participants in the class for 
inconsistencies in questioning. 


Exploring Information 

Searching relevant information is not just one step in the inquiry-based learning cycle, but an ongoing 
process in inquiry-based learning. My research showed that exploring related information may serve different goals 
at different phases of inquiry-based learning. For example, at the beginning, students explore information to find a 
workable topic, make an initial judgment on the difficulty of finding information related to their several potential 
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topics before they finally decide on their topic. If a student does not have much knowledge about their topic, then 
being able to find out good background information is often a necessary step before they can create some research 
questions. Once students have gathered initial information about their topic and determine their own prior 
knowledge, they can develop appropriate questions. . The next step is to search for information around their 
questions. Usually, the research starts from a broad search to a more specific search as the learner becomes more 
knowledgeable about their topic. 

This is a most sophisticated step in which research questions interact with the newly found information. 
Sometimes, the new information leads to discarding the old questions and creating new questions. Other times, 
students may realize that they need to refocus their research to determine answers to their questions. 

Students are very easily frustrated at this step. Online scaffolding is crucial. Online instructors should point 
out several things to students. First, the teacher should explicitly tell students search goals when exploring 
information. Also, the teacher should remind students to use their questions to guide their search instead of aimlessly 
searching through tons of information. Possible tools for this step include social bookmarking collections and 
sharing good background resources for students. 

Gaining New Understanding through Knowledge Construction 

One feature distinguishing constructivist from traditional pedagogies is its emphasis on knowledge 
construction instead of knowledge copying (Piaget, 1970). When a knowledge construction view is held: 

“Learning is basically a procedure of confronting a specific event, problem, or issue; acquiring and 
describing a body of information related to the event, problem, or issue; analyzing causal relationships; and 
stating explanations that are logically supported by the data.” (McCollum, 1978, p. 73) 

In our case, knowledge construction occurs while students explore information and make sense out of it. 
With the process of exploring more and more topic-related information, learners interact with those resources. 
According to my research, there are several types of interactions: 1) eliciting prior knowledge, 2) adding new 
understanding about the topic, 3) deciding what information that can help answer research questions, 5) identifying 
their knowledge gap, e.g., they found out they needed to know more about a topic in order to answer their questions, 
6) validating resources, 7) evaluating conflicting information. These interactions between a learner and the resources 
are essentially various mind activities that lead to new understanding. 

A good strategy for online instructors is to explicitly list the possible interaction types. They should also 
ask students to pay attention to the types of interaction, such as whether the resources can be validated and whether 
the resources can help answer their questions, etc. 

To facilitate such knowledge construction online, ask students to use blog or other Web 2.0 tools to 
document their knowledge, encourage them to refine their knowledge for any discrepancy between a claim and its 
supporting evidence, to reflect on the origin of that knowledge, and to organize it into a more coherent system 
(Coleman, 1998; de Vries, Lund, & Baker, 2002). In addition, social networks can provided opportunities for online 
discourse, because knowledge construction is socially situated and can be supported in a community (Lim, 2004; 
Topper, 2005). 


Presentation 

Presentation is usually the last step in the inquiry models discussed above. In this step, the knowledge gap 
is filled in around the inquiry topic. Students have discovered meaning within their posted inquiry questions, and 
solutions have been found to address the inquiry problems. Students are ready to present and discuss their findings. 
Research has found online communication through written text (email, discussion) often facilitates deep thinking 
and clear expression (Song, Singleton, Hill, & Koh, 2004). Therefore, in the online context, presentations can take 
various written forms instead of face-to-face presentation. Students can publish their inquiry findings on websites. 
Comments and feedback from peers or instructors can also add discourse and reflection (Rosebery, Warren, & 
Conant, 1992; Topper, 2005). In short, implementing presentations online through written forms and promoting 
discourse around the students’ presentations can foster reflection and critical thinking in the students’ online inquiry- 
based learning processes. 
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Table 1. Tools Used in Online Inquiry-Based Learning 


Inquiry Steps 

Tool Categories 

Tool Examples 

Selecting and 
narrow down 
topics 

Conceptual mapping 
tools 

Bubble: http://bubbl.us 
gliffy: https://www.gliffy.com/ 
idea Sketch: http://www.nosleep.net/ 
inspiration: http://www.inspiration.comtext 

Generating 

questions 

Social network or 
wiki 

Elgg: http://elgg.org/index.php 
Edmodo: https://www.edmodo.com/ 
Wikispaces: http://www.wikispaces.com/ 
PBworks: http://pbworks.com 

Exploring new 
information 

Social bookmarking 

Diigo: https://www.diigo.com 

Gaining new 
understanding 

Blogs and social 
networks 

EduBlogs:http://edublogs.org/ 
Wordpress:http://wordpress.com/ 
LiveJournal:http://www. livejournal.com/ 
Blogspot: http://www.blogspot.com 

Presenting 

findings 

Various publishing 
tools 

Wikispaces: http://www.wikispaces.com/ 

PBworks: http://pbworks.com 

StoryKit: http://www.youtube.com/watch?v=RSWQkUS4kXk 
AudioBoo:http://audioboo.fm/ 

DigitalStoryTeller: http://www.digitalstoryteller.org/ 
VoiceThread: http://voicethread.com/ 

Weebly: http://www.weebly.com/ 

Google site: sites.google.com 
Prezi:http://prezi.com 


Implications 

This paper is an effort to explore how to use different strategies and technology tools to support online 

inquiry-based learning. I hope it can expand our understanding about the important elements that constitute the 

online inquiry-based learning process and environment design. 
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Goal of Alien Rescue 

The goal of Alien Rescue (AR) is to engage sixth-grade students in solving a complex problem that requires 
them to use the tools, procedures and knowledge of space science and apply processes of scientific inquiry while 
learning about our solar system. Students, acting as scientists, are asked to participate in a rescue operation to find 
suitable relocation sites within our solar system for six different species of aliens who have been displaced from 
their home planets. Through inquiry-based activities, students practice a variety of problem-solving, self-directed, 
and collaboration skills using multimedia enriched cognitive tools. 

Target Audience 

Alien Rescue is designed for sixth-grade space science as a curriculum unit for about fifteen 50-minute 
class sessions. It is aligned with National Science Standards and the Texas Essential Knowledge and Skills (TEKS). 
Although the primary intended population is sixth graders, the use of this program, with proper instructional 
modification, has expanded to include 5 th through 9 th grade populations. Teachers have used it with students of 
various ability levels with success. From 2012 to present, the program is being used as part of the science curriculum 
by 16 middle schools in Central Texas with a diverse ethnic base. In addition, schools in at least twenty states (AZ, 
CA, CO, CT, FL, HI, IL, IA, KS, MD, MI, MO, MS, NM, NJ, OH, PA, SC, TN, WA) and three countries 
(Australia, Canada, China) have used and are using Alien Rescue. 

To assist teachers’ implementation in the classroom, a comprehensive teacher’s manual (over 160 pages 
delivered via a wiki-format) is provided to assist teachers in their implementation. The manual details PBL 
pedagogy, lesson plans for each of the 15 days, additional science content materials, and assessment tools. 

Key Design Features 

Alien Rescue delivers a playful experience in an intentional problem-based narrative. It combines game 
elements, play, and authenticity for the purpose of engaging students’ learning of science and enhancing student 
motivation. Authenticity is achieved by placing students in the role of young scientists and charging them with the 
task of saving distressed aliens. This central problem is presented through a compelling introductory video to create 
a sense of urgency. As scientists, the students are challenged to find new homes for the aliens by engaging in the 
process of scientific inquiry: identifying the problem, researching, forming hypotheses, testing and validating their 
hypotheses, and justifying their rationales. Thus, the problem-solving process requires students to think and act like 
scientists and communicate with each other, thereby demonstrating scientific literacy. In addition, students use a set 


503 


of 14 media-rich tools designed to assist learning. Learning therefore occurs as a result of solving a complex 
problem. There are six different types of alien species and each species has its own unique characteristics. There is 
not one single correct answer to the central problem. Some answers are more optimal than others. It is therefore up 
to the students to present evidence and justify their problem solution with a rationale. These complications present a 
challenge to sixth graders that encourages them to control their own learning path. 

This real-world scientific inquiry is coupled with a more playful experience and delivered through a 3D 
immersive, discovery, and sensory-stimuli-rich approach. When students enter the program, they are not given 
explicit instructions on how to begin problem solving. They must explore and discover the available tools, 
understand their functions and determine when to use which tool at the time. This design evokes uncertainty, 
mystery, and curiosity. The Research Lab (that houses Alien Database) presents information about each of the six 
alien species, including details about their physique, nutritional needs, and habitats. This detail is packaged into an 
interactive 3D tool that is designed to help establish a sense of fantasy. That is, the environment situates the learning 
experience through scientific inquiry in ways that help students learn the language of science through role-play in a 
science fiction fantasy setting. 


Performance Data 

All sixth grades from two public middle schools («=430) in a mid-sized southwestern city in the U.S. 
participated in a recent study (Liu, Rosenblum, Horton, & Kang, accepted). These sixth graders used AR in their 
daily 50-minute science classes as their curriculum for space science for three weeks. The findings using ANOVA 
with repeated measures showed sixth graders significantly increased their science knowledge scores after using the 
program. The average gain score from pretest to posttest for School 1 was 24.29 with M ma j e = 23.34 and M fema i e = 
25.78; and 13.3 1 with M ma i e = 12.28 and Mf ema i e = 14.46 for School 2. It is worth noting that for both schools female 
students had higher gain in points than their male counterparts. 


Students ’ Science Knowledge Test Scores 


Science Knowledge 
Score 

(% on 0-100 scale) 

Boys 



Girls 



Total 



n 

M(SD) 


n 

M(SD) 


n 

M(SD) 


School 1 

69 



45 



114 



Pretest 


57.17% 

(22.61) 


52.11% 

(13.76) 


55.18% 

(21.69) 

Posttest 


80.51%* 

(18.55) 


77.89%* 

(22.55) 


79.47%* 

(20.17) 

School 2 

134 



123 



257 



Pretest 


49.54% 

(21.81) 


41.82% 

(19.7) 


45.85% 

(21.15) 

Posttest 


61.82%* 

(19.18) 


56.28%* 

(16.28) 


59.17%* 

(18.03) 


* Significantly different from the pretest,/? < .01. Only those who completed both pre- and posttests were reported. 


In their open-ended responses, sixth graders stated what they had learned: about our solar system (the 
planets, moons, and their characteristics) (51% out of 515 units of responses); the scientific instruments (creating 
and launching probes and various instillments needed for each type of probe) (16%); alien species (8%); scientific 
concepts such as magnetic fields, gravity, and temperature scales (7%); problem solving (4%); conducting research 
(4%); managing a budget (2%); and working with others (2%). 

To compare students who used AR with those who did not, a study was recently conducted in a school 
district with two middle schools of comparable demographics. Alien Rescue was used with all sixth-graders in one 
school while sixth graders in the other school were taught the space unit in the usual way. Both schools have access 
to computers and the Internet. At the end of the unit, all students took a school district-created 15-item test on space 
science. The results showed a significant difference in the test scores between the two groups: F( 1,766),/? <.01, 
indicating the school used AR scored significantly higher. (This study is currently being written up.) 
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Mean % Score (out of 100) 

Scored 66% 
or better 

Scored 90% 
or better 

School did not use AR: «=384 
(female=174, male=210) 

80.11% 

89.58% 

26.82% 

School used AR: «=384 
(female=195, male=189) 

87.56% 

94.79% 

54.17% 


Attitude Data 

In another recent study (Liu, Horton, Kang, Kimmons, & Lee, in press), sixth graders were asked to the 
question: “How would you describe Alien Rescue to a friend?,” a total of 1,072 words were extracted out of the 358 
statements. The word “fun” has the highest frequency: 


Word Cloud 

Word 

Frequency (%) 

, INFORMATION^^ j 

Al 1 PM-0 EDUCATIONAL 1 1 , HELP - - 

jg BORING^ _I|,3 m6AME 

| ^SOLAR-SYSTEM 

B , kl LEARN 

J HOME 

fun 

learn 

solar-system 

aliens 

find 

helpful 

home 

information 

interesting 

probe 

game 

computer 

183 (17%) 
108 (10%) 
93 (8.7%) 
74 (6.9%) 
46 (4.3%) 
41 (3.8%) 
38 (2.7%) 
30 (2.5%) 
29 (2.7%) 
27 (2.5%) 
24 (2.2%) 
20(1.9%) 


The following comments, unedited, from teachers and students further showed positive attitude: 

Sample Comments from Teachers (2012 - present) 

• Alien Rescue (AR) was a fantastic activity. I've been involved with problem based learning since '95 and this is 
terrific way to include/integrate 21st century technology and skills. AR takes a problem, allows the students to 
ask questions and research and then propose possible solutions. The interaction and communication amongst 
the team members was terrific. It was great listening to them discuss/argue as they researched and collected 
info from their probes. It allowed them to learn about so many topics as they tried to find new homes for the 
aliens. Thanks so much for allowing me to use this in my class. I look forward to using this again next year! 

—Mr. P, Chartiers-Houston Jr./SR. High School, PA 

• I was amazed at how, after two weeks, all 125 of my students were still so drawn to Alien Rescue. They were 
having conversations about magnetic fields and elements on the periodic table in my classroom during lunch, 
and kids were going to the Library on their off-time to send probes and take notes in Alien Rescue! The lesson 
plans provided by Alien Rescue were rigorous, and all the background information I needed was available on 
the website. We will definitely use Alien Rescue in our Space Unit next year! 

-Ms. V, Four Points Middle School, Leader, TX 

• Alien Rescue is engaging for the students. It is a great example of problem based learning. The students must 
work as a team to solve a problem so it also encourage[s] collaboration. Our test and quiz grades are higher 
than the years before when we did not use Alien Rescue. I can tell they are learning the material because of the 
conversations we have when we debrief at the end of class. 

—Ms. C, Running Brushy Middle School, Leader, TX 

• I have never seen middle school students so engaged in reading and using data as when they were ‘playing’ 
Alien Rescue. When we were finished they asked if they could do it again because it was so much fun! 

—Ms. C, Loftis Middle School, TN 
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• My students have really enjoyed using Alien Rescue. Alien Rescue was a wonderful tool to integrate science in 
a fun, challenging, and innovative way. Alien Rescue embraces the idea of teacher acting as facilitator as 
opposed to teaching. Alien Rescue provides the circumstance that definitely keeps the students engaged while 
the teacher facilitates the learning environment. 

—Ms. A, Columbia Local Schools, OH 

Sample Comments From the Sixth Graders (2012 - present) 

• I like Alien Rescue much more than other science activities because we’re not just sitting at our desk doing 
work that must be done on our own, it’s a fun activity that ties in with what we’re learning. 

• Alien Rescue was better than other activities because I liked learning about the different things. Alien Rescue 
gave us a chance to work independently on a project by ourselves. I also liked that we could work with different 
people. Collaboration caused us to debate and come up with more correct answers than if we were working by 
ourselves. 

• Because Alien Rescue you can learn what scientist really do and how they learn about all the planets. 

• I liked Alien Rescue more than other science activities because it was a group project, we got to do it on the 
computer, and it was like a video game . 

• 1 liked doing Alien Rescue more than other activities because you get to do hands on activities. I find it more 
interesting than reading out of the book, it would prepare me to be an astronaut ! 

• Alien Rescue is educational, but at the same time interactive and fun, like a video game. You are also much 
more independent in Alien Rescue. 

Videos of what teachers and students say are available at: http://alienrescue.edb.utexas.edu/feature videos.php 

Our Development Model 

Alien Rescue team consists of a group of graduate students working collaboratively under the supervision 
and guidance of faculty. Harnessing students’ diverse talents and ideas is a major characteristic of our development 
model. Students’ engagement in the project is largely driven by their interest in creating a quality technology 
program to enhance learning while developing multimedia production and research competencies. Through this 
process, students have gained valuable software design and technical development skills. This has been a training 
platform for future designers and developers, instructional technologists, and educational researchers. 

Access to Alien Rescue 


To access Alien Rescue: 

1. Install the Unity Web plugin (free): http://unitv3d.com/webplayer/ 

2. Log in to: https://alienrescue.net 

3. Use the Registration Code: 

(When you use Alien Rescue for the first time, click the "Click here to register" link. On the registration page, you 
can create your own username and password with the unique code we provide you for the Class Code. Once your 
account has been created, you will be able to login to use it.) 
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Abstract 

This paper focuses on reading Chinese film through visual literacy. Students at the University of Rhode 
Island have been learning visual literacy to read Chinese film since the spring semester of 2013. They explore 
Chinese film from historical, social, economic, political, and cultural perspectives. Students develop familiarity with 
the ways China has addressed modern problems and its links with the world to broaden their views and enrich their 
understanding of the world through film and film industry in China. 

Introduction 

In response to the rising Chinese film market and the dramatic shift of dominance of Chinese film making 
industry, FLM495 Chinese Hollywood — Through the Lens of Visual Literacy has been offered by the Film/Media 
Program of the Harrington School of Communication and Media, University of Rhode Island to meet the needs of 
students and educational enterprise since the spring semester of 2013. 

As China’s economic forces are moving forward, Chinese film industry is coming to Hollywood or Chinese 
Hollywood is developing to compete with the film industry in the US. This research project studies visual literacy 
training to explore Chinese Hollywood. Students study Chinese film from historical, social, economic, political, and 
cultural perspectives. They are engaged in a learning experience for critical and cultural analysis of Chinese film as 
visual texts. Students develop familiarity with the ways China has addressed modern problems and its links with the 
world to broaden their views and enrich their understanding of the world through film and film industry in China. 

“What is Hollywood?” is one of the first definitions that students faced to define. In addition to its birthday 
place of American film industry located in a district of Los Angles, Hollywood has shaped the US culture. 

Hollywood stars become idols of many American film viewers because of their social status, privileges, and world- 
wide recognition, and being rich and famous. 

In searching for a definition of Chinese Hollywood, the US Hollywood definition is distant from the 
content or substance to define Chinese Hollywood. There is no single film studio(s) in one location in China to 
identify as China’s film industry birthplace to produce its domestic film. Since 1996, Hendian, the world’s largest 
outdoor film studios in China, located in Zhejiang Province on a 650-hectare land, equivalent of 1,600 acres, started 
film production in 1996 (Shao, 2013), “Chinese media claim that, including advertisements, 30,000 productions 
were shot here last year. The studios boast, among many vast sets, a full-scale replica of Beijing's Forbidden City.” 
(The Economist, 2012). Los Angeles Times reported that the Chinese largest film studio will be built in Tianjin by 
the Seven Stars Entertainment investing more than $1.27 billion. This new entertainment and media complex outside 
of Beijing will be called "Chinawood." (April 30, 2012). While the world is waiting for this “Chinawood” to be born, 
Los Angeles Times reported that “China's richest man, Wang Jianlin, announced to build what he is touting would 
be the world's largest movie studio investing $8.2 billion. The project will be named the Oriental Movie Metropolis, 
the development is to consist of 20 studios and a permanent underwater stage. It is to be located an hour outside 
Qingdao, a coastal city with more than 8.7 million people.” (Los Angeles Times, September 22, 2013). In addition, 
it is predicted that “China will top U.S. as the biggest film market in the world by 2020”, (Tsui, 2012). It can be 
concluded at this point that China’s Hollywood location is in a constant changing status as China reshapes itself. 

The meaning of establishing this physical Chinese Hollywood is changing and evolving as well. China and the 
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world are witnessing this physical location or birthplace or identity of Chinese Hollywood to be envisioned, planned, 
blueprinted, formulated, built, created, established, executed, and realized. Does the location signify the importance 
or significance of a country’s film industry or a new beginning of Chinese film production? In addition to its 
symbolic meaning of economic and cultural power and strengthen, what type of opportunities do the world largest 
studios provide for China to develop film industry and open its market? Will these studios attract the best producers 
and directors in the world to come to China to make film? Where will be the film market be? How will this physical 
Chinese Hollywood shape the Chinese culture? How will the educators prepare students for this opportunity and 
audience to receive the new wave of film from his new production line? 

From historical, cultural, and global perspectives, the physical location of Chinese Hollywood may not 
matter as much as that the US Hollywood does. Naming Chinese Hollywood presents multifactor challenge. What 
name should be to best define Chinese Hollywood to embrace its people, culture, civilization, heritage, history, 
economics, politics, society, and global inclusion? The following possible names are coined to name Chinese 
Hollywood: Chinawood provides the identity of China and to rhyme with Hollywood and Bollywood, but it also 
rhymes with the word “wood” in relation to a tree. Chywood or Chiwood is also possible in sound, Huawood or 
Hualllywood is to embrace the rich Chinese civilization and culture in the world. “Hua” means Chinese civilization 
or Chinese culture. From author’s perspective, Huallywood is the preferred name. For this article, Chinese 
Hollywood and Huallywood are used interchangeably. Film about China made in China, Hong Kong, Taiwan, US, 
and elsewhere in the world will be under the umbrella “Hua” meaning “Huaxia”or “Zhonghua” meaning (China) to 
show Chinese culture through different types of filmmaking styles, cinematic techniques, and semiotically coded 
visual texts about Chinese culture. In essence, Huallywood, Chinese cinema, is a representation of Chinese 
civilization, culture, heritage, philosophy, history, politics, and contemporary daily life of the Chinese in the world. 

If Hollywood has been shaping the US culture beyond its geographic and cultural boundaries, it has been 
influencing the world culture by its film production and its film stars, has Chinese film and its film industry been 
reaching the same purpose? Is China getting ready to build this Huallywood in one physical location to symbolize 
the ideal of Huallywood to dominate the film industry and its market in the world for its culture and influence of the 
world through its film? Hollywood has been made famous for its own film making system/history, identify, culture, 
values, celebrities, Oscars, and Golden Globe Awards through its film of everyday American life. Will China follow 
the same pattern or ride a different path to lead its Huallywood? Hollywood is a global stage and a symbol of 
recognition in acting, directing, and film making. Will China establish a Huascar Award to attract the brightest and 
the best in film making industry in Huallywood? 

Hollywood stars become idols or heroes of many American film viewers. Hollywood stars became US 
President or Governor of the State of California. Many dream to become Hollywood stars for social, economic, 
cultural, and even political status. Throughout history, actors/actresses or Chinese performers in general of different 
ability, prestige, or different performance types are socially ranked low in status until recently some changes have 
taken place. Acting is one of the least pursued careers for young people in China. Western influence has changed 
this tradition, however, the deep-rooted culture remains. Culturally, to pursue knowledge to become a scholar or a 
learned person is a much more superior career path for many Chinese. (Since Chinese character are impossible to 
insert here for this writing, the phonetics of this Chinese saying is as follows: wan ban jie xia pin,wei you du shii 
gao). How China will promote, encourage, and enhance the social status of Chinese film stars and performers of 
any kind will remain to be examined and studied. 

Hollywood has its own system to select the best script for film. China has loosened its policy of reviewing 
screenplays for film. Censorship is at the government’s decision on how each film is made or how each script to be 
chosen. It is intended to safe guard the ideological structure for the education and moral standards of the country in 
addition to its political stand. It is an art to study Chinese politics and culture to gain confidence and acceptance for 
film directors who wish to produce film in China or for the Chinese’s film market. 

All these topics and issues were addressed in the spring semester of 2013 when the class of Chinese 
Hollywood-Through the Lens of Visual Literacy studied twelve Chinese film and analyzed them. One of the film 
Shanghai Women (2001) by Peng Xiaolian is used for this research to illustrate how to read Chinese film. Peng 
Xiaolian began her film career at the Beijing Film Academy and is a member of China’s famous “Fifth Generation” 
of directors. After graduation she directed several successful films including Me and My Classmate (1986) and 
Women ’s Story (1989). Peng furthered her film studies at New York University, and after earning her Master’s of 
Fine Arts from NYU, she returned to Shanghai in 1996 to make film. Peng has directed many feature films, 
including Once Upon a Time in Shanghai, which was released in the United States, and Shanghai Women, which 
was highly successful in Japan. One of her latest films, Shanghai Story’ (2004), received four Golden Rooster 
Awards, China’s equivalent to the Oscars for best picture, director, actress, and supporting actor. She also filmed 
Shanghai Rumba in 2006 and Storm under the Red Sun in 2010. Peng's Shanghai Women delves into the lives of 
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three generations of women in contemporary Shanghai. It is required to watch this film to understand the content of 
this analysis. 


Research Methods 

Research methods from social sciences such as surveys are used to collect data on the responses from 
viewers in the US and China. The research design consisted of a structural analysis for encoded message and a post 
structural analysis using reception theory to analyze the how viewers are positioned by the visual text ( Shanghai 
Women) created by the director. 

Focus Questions 

1 . Why do the Americans and Chinese like or dislike this film? 

2. Who is being addressed by this film? 

3. How does this movie position its viewers? 

4. What are the cultural codes in this film? 

5. What meanings are constructed when viewers interact with Shanghai Women ? (Chinese viewers vs. 
United States viewers). 

6. “What is the intent of the director in making this film? 

7. “How do the viewers interpret the cultural codes in the film individually and as members of a 
community that have access to those signs and symbols?” 

How does Chinese film director Peng Xiaolian construct their subject? Or, whom does the film director 
think their viewers are? These questions have not been studied in the recent Chinese film research fully to invite 
future film directors to prepare themselves for their film making experience. The purpose of the study is to apply 
reader-response criticism to examine the meaning-making in reading Chinese film by US and Chinese viewers. It 
investigates the relationships between and among directors, texts and viewers. The study ascertains the intentions of 
the film director; examines the meanings made by the viewers through surveys; compares the similarities and 
differences of director' intentions with their film; and compares the similarities and differences between the 
director’s with views of the viewers. 


Theoretical Background 

Reception theoiy is used for this study. It was originally a German reader-response theory (Jauss, 1982). 
Reception theory is an umbrella term used in this article to include terms such as reader theory, reader-response 
theory, reader-response criticism. They are used interchangeably in this article. For this study, the literary reading 
theory is appropriated. Structuralist theorists believe that the meaning of a literary work resides only in the text (i.e., 
the text has coded messages given by the author). Writers of structuralist theories consider that the reading process 
is the participation of the reader with what is intended in the text. The history of reader response theory is shown in 
the works of Hans Robert Jauss (1982), Roman Ingarden (1973), Stanley Fish (1980), and Wolfgang Iser (1978). 
These theorists emphasize the relationship between reader, text, and author. Unlike structuralist theorists, "all 
reader-response critics focus on readers during the process of reading" (Mailloux, 1982, p. 20). American reader- 
response theorist Stanley Fish departed from the phenomenological model of reader theory developed by Jauss to a 
social model. Conceptualizing that meaning is created by the reader in the reading process, Fish extended his theory 
by examining the social construction of knowledge, which influences the reader's subjectivity. His concept of 
"interpretive communities" provides a basis for analysis of readers of different communities who share a community 
of interests. Reader-response theory "replaces examinations of a text in-and-of-itself with discussions of the reading 
process, the 'interaction' of reader and text" (Mailloux, 1982, p. 2). It examines "the role actual readers play in the 
determination of literary meaning, the relation of reading conventions to textual interpretation, and the status of the 
reader itself' (Tompkins, 1980, p. ix). De Vaney (1991) explains the difference between post structural reader 
theories and structural theories as 

post structural reader theories developed as a reaction against structural analysis in which the media text 
itself is of dominant interest. Structural analysts believe that messages are encoded in the media text and 
the viewing process is simply one of decoding a fixed message.... If one considers the act of 
communication, the media text is only one element in that process. In any communication, messages are 
created by a sender or author, encoded in a text and differently decoded by readers or viewers. Therefore, 
the sender's or author's intent, the text and the reader/viewer become important parts of the analytical 
equation (p. 2). 
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Post structural reader-response theory has been applied to examine the relationships between and among 
author, text, and reader of educational television programs (De Vaney and Elenes, 1991), computer programs (De 
Vaney, 1993), hypermedia (Kirby, 1993), and postmodern art work (Ma, 1993). De Vaney emphasizes that a 
reader/viewer creates meaning when interacting with a media text that is coded and in which the author's intent is 
embedded (De Vaney, 1993). In order to adapt reception theory for the analysis of Chinese fdm, the following 
definitions of concepts are provided in the hope that they may help explain the theory and its adaptation for the 
study. (Ma, 1995) 

Reading , in this study, is the process of creating meaning while interacting with the text, which is the Chinese film, 
Shanghai Women. It is the reader/user who creates meaning. 

Text , in this study, is defined as a communication that has been constructed by the directors. It is Shanghai Women. 
Viewers , in this study, are defined as those who participated in viewing the film selected for the study. 

Director , in this study, is Peng Xiaolian 

Communities of viewers refer to a group of people who share similar social, cultural, economic and other positions. 
Sixteen US university students and 10 Chinese university students participated in the survey of this selected film. 

Semiotics and cultural codes are analyzed for the study. Semiotic is the science of signs. It studies the 
relationship between form and meaning. Signs are organized into systems of meaning and recognized by codes. 
“Signs are assigned meaning based on historic patterns of use that are recognized within social/cultural groups. 

These patterns of use are called codes” (Pomper, 1988, p. 18). A sign comprises two types of meanings: syntagmatic 
and paradigmatic meanings. Syntagmatic meaning refers to the meaning that is assigned based on syntax, or based 
on the relationships among signs. Paradigmatic meaning is derived from other systems or codes. For example, color 
white is often used to symbolize purity in Western culture. Brides wear white gowns. While in Eastern culture, color 
white symbolizes sorrow and sadness and it is associated with death. Brides in the Eastern civilization wear red, 
which conveys the meaning of celebration, good fortune, happiness, and prosperity. Flowever, color red often 
suggests danger and stop in Western culture. (Ma, 2006). Film, television program, and visual images are visual 
texts. They all bear culturally and socially constructed codes, which shape the meaning of messages conveyed in a 
visual text. Viewers of a community have access to those codes of signs and symbols. DeVaney (1991b) referred 
this code as a syntax of the organization or arrangement of a communication whether a sentence or a visual scene. 
When an arrangement is consistently repeated in the production of a communication, a code develops (DeVaney, 
1991a). Another example of visual syntax in most of the Chinese films is thunderstorms and heavy rains, which 
suggests something tragic is going to happen. In film or television programs, another example of code is code of 
realism (DeVaney, 2001). She reminded us that camera and structural codes are used to imitate reality. (Ma, 2006) 

Analysis 

Responses were collected from 19 viewers who are university undergraduate students at the University of 
Rhode Island and the School of Journalism and Communication, Southwest University of Political Science and Law, 
China. The online survey questionnaire consisted of ten open-ended questions, of which seven contained scenes 
from the film. 

The first question asked was: In your opinion, why do Americans like or dislike the film "Shanghai 
Women"? 

Responses from the US students are varied: 

• “I think that Americans would not be a fan of this movie. I feel this way because of this film's narrative 
structure. To me it doesn't appeal to the American taste that they want in cinema.” 

• “It has a universal, multicultural story plot that is easily recognizable in most families that has dis- 
functions.” 

• “They dislike the film because it shows the rough sexist lives that many middle class Chinese [Scic.] 
women endure. This does not agree with american [Sic.] audiences because they support more feministic 
equality.” 

• “I think Americans would really like the film because it touches on some topics that are very relevant in our 
culture like divorce and raising a child as a single parent. The acting was good and the film was made in an 
American [Sic.] style.” 

• “I don't believe that this film would appeal to most americans [Sic.] as it is largely culturally based and that 
can be difficult to relay to an audience that is not completely aware of the subject.” 
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• “I don' t think Americans like "Shanghai Women" because it's a film that displays the hardships of Chinese 
women in capitalist China. If you're studying Chinese or have an interest in the country than I suggest you 
watch this film.” 

• “I think Americans will like Shanghai Women because it is a new movie that has a very interesting story 
about a woman and her daughter. Divorce and a woman's independence are things that are considered much 
more normal in the Western culture than in China even to this day and therefore I think that a lot of the 
American public will resonate with the issues raised in the film. We also know that Xiaolian Peng studied 
in the United States at New York University and could more easily take certain aspects of American culture 
and implore it in a movie like Shanghai Women.” 

• “I think American's would dislike "Shanghai Women" because it is relatable content, especially with 
women that relate to the feminist movement.” 

• “I think that Americans would like this film because of the way the director handles the setting of Shanghai. 
She takes deliberate care to make the city as much of a character as the actors on screen.” 

• “I feel that Americans would enjoy this film. I think that the story is interesting enough to capture their 
attention and keep them locked in.” 

• “I feel that the stereotypical Americans would dislike this film. Their biased nature could be negative 
towards this feminist themed film. Not to mention the foreign nature of the film. Many cannot relate to the 
specific topics this film hints upon. 1 think it is pretty general to say this, but I believe most would not enjoy 
this flick.” 

• “I think American's would like this film because it shows what the lifestyles of people living in Shanghai 
are like.” 

Responses from Chinese students are as follows: 

• “I think Americans would love this movie, because it represents a true China last century... [It will give] 
help to increase the understanding of Chinese Americans” 

• “Westerners’ impression of China is about Shanghai Qipao, a symbol of the Orient. This is a different film 
for the foreigners to enjoy.” 

• “Cultural differences may attract Americans.” 

• “I feel that the Americans will like this film because it touches the true essence of humanity and life.” 

• “The film shows the typical ‘Shanghai Women. The story is not attractive.” 

The second question asked was: Why do you like/dislike this film? 

Responses from the US students are as follows: 

• “I like this film for the exact reason why American's wouldn't like it. The narrative and the voice it carries 
within it are truly amazing. You can see two worlds colliding together, and that is what makes this such a 
great film.” 

• “Because it reminded me of my family and how we can disagree.” 

• “I like the film because of the realism behind it. The film focuses on a true depiction of the female lifestyles 
of women in contemporary China. This provides insight into the lives and cultures that I would otherwise 
never be exposed to.” 

• “I liked this film because it dealt with subject matter I can relate to, like divorce and being a kid who wants 
to be on your own but your too young to break away from your parents. The film also was great because of 
the acting and the overall production quality was good. ” 

• “I'm interested in Chinese culture so I enjoyed the film. It's a character study which interests me.” 

• “I like the film because it is personal to me. My parents got divorced when I was 1 1 or 12 years old and I 
know what it's like to have to go through a bad divorce, let alone two divorces. I know what it's like for a 
woman to feel unhappy in a marriage and I am glad that the mother didn't feel like she needed to be with a 
man to be happy. Also, I like that fact that Chinese people learn to be independent at an early age. My 
mother worked as a teacher for many years as well so the conditions in the movie also became personal to 
me.” 

• “I enjoyed the film because it gave an interesting look into the sexism that goes on in China, even in the 
present day.” 
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• “I personally like the film because it was visually stimulating. There was always a lot going on within the 
frame that helped to reveal more about the characters, and given that it's a Chinese film it does help for the 
visuals to be there and comment on the action.” 

• “1 personally liked this film for the exact reason 1 listed above. The story really captured my interest and 
brought awareness to my eyes.” 

• “1 don't think 1 liked or disliked this film. There were some interesting scenes and topics and also parts 
which 1 think could have been revamped. 1 understand and recognize this is an older and eastern fdm, 
which makes it harder for me to relate to, but 1 like to think 1 am more open minded than the average 
‘American’.” 

• “1 liked that it was contemporary and dealt with real life issues (the film was realistic)” 

Responses from Chinese students are as follows: 

• “1 do not have much feeling about this film because it has too much elements about Shanghai, ft is the 
Shanghai in the 30s.” 

• “1 like this film because it truly portrayed the hardship of life and the fascination of human nature.” 

• “1 like this film because it provides me with knowledge of women in Shanghai and their lives for our 
generation. These women images are quite in contrast with stereotypical Shanghai women who are elegant, 
composed, and shrewd.” 

Scene #1 What does this scene below remind you of? 

The author selected three scenes from the film with viewers’ responses for this article due to the limited 
space of this publication. 



Responses from the US students are as follows: 

• “This reminds me of a shot of some place in New York which always the mind to wonder. Even though 
this film is taking place in Shanghai, this shot can be seen as breaking down the cultural and spacial 
barriers around it. Refocusing the aim to an any where kind of place. Which in the end, we can re-divert 
the location and place it in some place closer to us, which then will make us feel more strongly for the 
emotions dwelling inside the film.” 

• “The urban areas of most big cities.” 

• “This reminds me of cramped slums that resemble the tenement housing in early New York City a long 
time ago. The cramped conditions and poor economic standard of living is something that all people 
globally can identify with.” 
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• “This reminds me of the tight living spaces in the cities like new York and Philadelphia.” 

• “I love this shot, it reminds me of most movies that placed in high-rise apartments in low income areas, my 
favorite being Hitchcock's Rear Window. It's great because it allows you to see how involved everyones 
[Sic.] life is, whether they want it or not.” 

• “It reminds me of the slums that I would see in New York when I grew up there. Very similar to Shanghai.” 

• “It reminds me of how they lived and how poor the conditions were. It's interesting to see that there are so 
many people who live in tight spaces. Five or more people live in a one-bedroom apartment. But then again, 
just because Americans like a lot of space doesn't mean that Chinese people necessarily do. 1 think this is 

an interesting comparison between Chinese, especially Shanghai, and American culture.” 

• “This scene represented the poverty of shanghai. It also reminded me of the early immagration [Sic.] 
period in America, with very small, impoverished, tightly packed and communities.” 

• “This scene reminds me of a Martin Scorsese fdm, any one will do. All of Scorsese's fdms are pretty much 
set in New York City and this shot reminds me of Little Italy. There's a lot going on in the shot but there's a 
lot more going on just behind what we see in the shot, it's very interesting.” 

• “This scene reminds me of New York City. The way the tall buildings tower over head, with the railings 
add that feeling of cluttering. Growing up this image was always associated with the ever hectic New York 
City.” 

• “It reminds me of the projects and a lower income part of society. Many movies have these kind of settings, 
which instantly enduces a stereotype on the character. Many people see this as a slummy and sketchy place 
to be. But like everyplace, there can be a diamond in the rough. This place reminds me of the setting of 
"Cinderella Man". 

• “This reminds me of the living conditions in Shanghai” 

Responses from Chinese students are as follows: 

• “It is a true picture of that time.” 

• “The crowded residential housing.” 

• “It reminds me of the lively, complex, and memorable Chinese city life in the past.” 

• “The current housing shortage condition can be traced back to the past.” 


Analysis 

Intertexuality refers to meanings are created for one visual text based on prior texts of an individual. The 
author defines that visual intertexuality invites viewers to construct meaning while interacting a visual text based on 
prior existent texts. Community of viewers share the same meaning construction process to create similar meaning 
through visual intertexuality. 

The cultural code of this scene that this is a shot of ordinary apartments in Shanghai in 2000. Housing was 
an extreme difficulty issue in Shanghai for many residents. 

Some US viewers responded that this scene reminded them of the slums in New York based on their prior visual 
texts and cultural experience. 

Chinese viewers by their age responded that this scene reminded them of the housing situation in the past. 
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Figure 2 

Scene #2: What Does This Scene Remind You Of? 



Responses from the US students are as follows: 

• “This scenes reminds me of rather uncomfortable things. It is too forced and almost too appropriate. By 
that I mean, in Chinese culture it seems that being married is a social value, like it is looked upon as a 
symbol of success. So in this marriage it is almost as if you can feel the tension building and this is a great 
example of it.” 

• “An American wedding ceremony.” 

• “This reminds me of a lot of weddings in the US with the bride and groom taking their wedding photo. But 
seeing the daughter with her head down kind of dejected from the rest of the characters reminds me of the 
pain children go through when their parents remarry.” 

• “This reminds me of an American military wedding. White dress, uniform. I'm ignoring the fact that its not 
an actual wedding. ” 

• “Traditional Chinese wedding. Very similar to American Weddings. The women on the left looks very 
depressed, yearning for love.” 

• “The mother has remarried because she thinks it's better for her daughter and a better life to be with a man.” 

• “This scene was very similar to an American style wedding.” 

• “Culturally it reminds me of the facade that marriage is made out to be in American culture. If we look at 
shot composition, here we have these two posing for the wedding picture and we have this women in the 
bottom left hand side of the frame in the dark. In America weddings are this big showy event so it's 
interesting to look at this shot in context of Chinese culture. In China marriages are celebrated and honored 
so here we have these two posing and making a big "to-do" about their wedding picture and in the corner is 
this girl who is all alone and in the dark.” 

• “This is a very interesting scene. These two people were just married, but the clearly have no feelings for 
each other or any interest in each other.” 

• “This reminds me of an arranged marriage that is commonly found in eastern culture. Although this one 
differs in plot, you can see the similarities throughout the scene. The lighting of this scenes gives me an 
eerie feeling. It doesn't give me the warm and fuzzy feeling that a normal wedding would. I feel like the 
director did this on purpose.” 

• “This reminds me of a non traditional Chinese wedding.” 

Responses from Chinese students are as follows: 

• “It is a wedding photo of that time.” 

• “Middle-aged people have wedding dream.” 
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• “At that time, people did not have much money. But, a wedding photo was still a must because a wedding 
photo represents a testimonial of a couple coming together. And it was a testimonial of the time.” 

• “The people of Shanghai have adopted Western-style wedding style.” 


Analysis 

The cultural code of this scene that this was a shot of a wedding photo in Western style with the teenage 
daughter sitting on her chair waiting. 

Most of the US viewers responded that this scene reminded them of the American wedding. 

Chinese viewers responded to confirm the importance of having a wedding photo and particularly a western 
style wedding photo. 


Figure 3 

Scene #3: What Does This Scene Remind You Of? 



Responses from the US students are as follows: 

• “This scene reminds me of women collectively and through out the ages in China, in this shot we have 3 
different views of Chinese women, older, middle-aged, and younger. All having experienced or 
experiencing what it is to be a woman in China.” 

• “American, multigenerational family interaction.” 

• “The women are all together, many generations of the same family are discussing the proper ways of life, 
ways that the young girl doesn't agree with and she is more of a free spirited individual.” 

• “This reminded me of how awkward living in with others in your family can be when you have no privacy. 
And seeing how their conversations in the home were with everyone who could hear, even people on the 
other side of the wall.” 

• “This scene is exactly as the what [Sic.] the film is about. These are your Shanghai women. Every 
generation in one room.” 

• “Women in Shanghai develop close relationships when their spouses worry about making money instead of 
focusing on their marriages/relationships.” 

• “It reminds me of when the mother has left her second husband and marriage and gone back to her mother's 
house. Her sister-in-law is talking down at her and making her feel badly about leaving another man.” 

• “This dinner represented the structure of Chinese families. While the mother was disciplining her daughter, 
the Grandmother chimed in, and the daughter submitted to the Grandmother.” 

• “This scene reminds me of the movie Jerry MaGuire when the women are all sitting around Renee 
Zellweger's home talking about the "perfect man." This seems to be very reflective of that in terms of the 
women giving advice on marriage or how to pick and choose a man, etc. The woman standing even looks 
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like Renee Zellweger, in a way. There seems to be an image that, no matter what nationality you are, 
filmmakers like to keep with and not stray far from as far as the leading lady.” 

• “The feminist encounters and discussions are some of the main parts of this film. They show the 
relationships and bonding of the characters, through their interactions. There are plenty of American 
movies like this, which aren't necessarily feminist in nature.” 

• “This scene reminds me of how close all the family members are (as far as living together goes with the 
tight spaces) & 1 feel that this is what this particular shot is incorporating (with having all of the women in 
the house close together in one single shot.” 

Responses from Chinese students are as follows: 

• “Shanghai women.” 

• “Gossip among women.” 

• “It is a profound reading of family relationship. It is hard to explain it, but you can feel it by heart.” 

• “Shanghai all ages women dress style.” 


Analysis 

The cultural code of this scene that this is a shot of the four women in the film. It invites an analysis to 
study the film angel, position of each woman, their facial expressions to study the power and relationship in this 
family/story. 

The US viewers provided such an amazing analysis using visual intertexuality! 

Chinese viewers by their age responded to this scene with cultural experiences and stereotypical depiction of women 
in Shanghai. 

What is the intention of the film Director? 

The director’s intention of this film is to portray the values of three generations of women in Shanghai or in 
China. Throughout history at anytime, women are constantly seeking their own space in life. (Peng, October 24, 
2013) 

Meanings are constructed out of this visual text in conjunction with the socially situated viewers. Viewers 
created meanings when they interact with the visual tests through visual intertexuality. Viewers’ prior knowledge 
and experience shape their meaning creation. Community of viewers of US cultural background share similar 
interpretations of this film. So does the community of viewers of Chinese cultural background. 

Conclusions 

Chinese Hollywood, Huallywood, Chinese cinema, is a representation of Chinese civilization, culture, 
heritage, philosophy, history, politics, and contemporary daily life of the Chinese in the world. Historically, Chinese 
film is art for art’s sake. Filmmaking is an art rather than solely for the market. Film is part of the government’s 
overall program. Private film companies are few and they are new enterprises. Film studies majors are available only 
in specialized film academies for the most talented and brightest students. It is not a regular major in comprehensive 
universities. Market is not a priority for filmmakers. Market is always there domestically as the film is made for 
predetermined purposes. How will this change as China enters the market driven film world? If China is going to be 
the largest film market in the world before 2020, how to prepare and educate film making directors for this market 
will become a pressing educational agenda. At the same time, if Chinese film industry is going to seize the film 
market in the world in the future, training, orienting, and educating audience will become an educational enterprise. 
Note: The author would like to thank the students in FLM495 Chinese Hollywood-Through the Lens of Visual 
Literacy for their participation in the film survey. 
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“ENVIRONMENTAL EDUCATION FOR 
PRIMARY EDUCATION (2013)” 

Nadiroh and Suwirman Nuryadin 

ABSTRACT 

The Guidance of PK and LK (particular education and particular services) about environmental education 
policy (PLH), committed on February 19, 2004 by 4 (four) Departments, the Ministry of State for the Environment 
(KNLH), Ministry of national education. Religion Ministries, and Ministry of Internal Affairs. 

To sketch deeply about the program which can be supportedly applied there are some recommendations 
used, such as: (1) redirected to the Ministry of education and culture, to keep doing, socializing and reinforcing and 
revitalizing the ideal informas policy of UNESCO, and (2) to principals and teachers of all subjects/ class teachers, 
in regular schools, inclusion as well as exceptional schools, performance spots for kids in particular needs because 
of marginal social attempts, to continue to undertake the mandate responsibility professionals respectively, so that 
the raised knowledge and skill in the performance of particular education and particular services with reference to 
the PK Guide and LK about Environmental education (EE) as a result of this matter. 

I. Backgrounds 

If the country is permitted to drag on us will be difficult to follow in an effort to reach Millennium 
Development Goals (MDGs): (1) eradicate extreme poverty and hunger, (2) achieve universal primary education (3) 
promote gender equality and empower women (4) reduce child mortality, (5) improve health mother, (6) combat hiv 
aids malaria and other diseases, (7) ensure sustainable development and environmental conservation and, (8) 
develop a global partnership for development (United Nations Development Group, 2003). To Indonesia this 
objective can be achieved if supported by the people and nation or quality benefit of Indonesian human resources 
who have skill, creativity, independence and highly competitiveness. 

The right to education is sustained by a global treaty about World Education Forum in Dakar, Senegal, 
2000, about poverty is expected to reach EFA by 2015. The Attractiveness of this harmonious with spirit and soul 
article 31 constitution the right of every person in obtaining education, UUSPN No 20, 2003, article 5 paragraph 1: 
“every citizen has the right to have a quality in education”, article 12 paragraphs 1: “every student entitled to get 
educated according to your interest, talent, and ability”, chapter 32 about special education and special education 
service. 

Government Regulation No 17/2010, concerning settlement organizationin education, section VII of 
organization particular education and particular educational services ( PK and LK ), part of the union, general: 
article 127: particular education is education for learners has the difficulty in attending a lesson because of 
disorder in physics, emotional, mental, social, and / or has the potential intelligence and special talents. Article 53, 
the government has responsibility to give college fees and / or assistance for free / special service for children from 
poor, refugees children, and kids who lived in remote areas. 

Regarding upon the policy, the need PK and LK that could hand the real life internally protege and 
external in the shape of the natural, social and physical environment. While the principles of it is not 
available. Therefore, it is needed PK and LK principles in the field of Environmental Education spread all over of 
Indonesia, as a device for principals and teachers in taking care of PK and LK for children aged 6-15 years emerged 
from marginalized communities (Social disability) either at school that has been designated by the country as a 
school Inclusion or not, however inside the class, there is a number of those students from marginalized 
communities. 


II. THEORY STUDIES/ CONCEPTUAL 

A. Environmental Education 

Environmental Education (EE) is a multi dynamic concept that deals with a new vision of education that 
work against empowering persons of all levels to take in charge in creating an environmental conservation in the 
near future. EE is an attempt to change the custom and life style for the public transport. The values that require to 
be enhanced in a sustainable development through education to change custom and lifestyle for positive public 
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transport are as follows: (1) Respect the values and rights of all human kinds across the planet and Earth and an 
agreement to social and economic justice for all; (2) Appreciate the human rights of future generations and a 
commitment to responsibility between generations; (3) Appreciate and care for the community of life with which 
includes protection to and improvement of the ecosystem of the planet Earth; and (4) Appreciate cultural diversity 
and commitment to build local and global culture of tolerance, peace and Nonviolence 
(http://www.UNESCObkk.org/index.php. id=3808). 


Environmental Education 

+ 


Educational Cone ept 

Living Environment Concept 
+- 

Sustainable Development Concept 


+■ 

Environmental Care Concept 




Output 

0 


Natural Intelligence 

Knowledge/Comprehension 

Affection/Care 

Eco Friendly Behavior 


B. Children in need of social and psychology (Barrel Disability) 

A guide for special education and special service held more focus on target special services to disorder; 
social child caused by various factors, including social marginal victim of narcotics and drugs, and mindset that 
results in extremely mental disorder as well as other factors. Other related learned by a variety of other schools 
generally, usually educated in standard school, studio, and the workshop of inclusion. 

There are three kinds that urgently required to be put at a glance about the learners of inclusion school, 
such as: (1) comprehend learners who are not the same and/or have the great intelligence and particular talents, (2) 
the personality/special needs for learners, and (3) the level of intelligence. As for the attempts have as follows (1) 
who are blind or visually impaired; (2) the hearing or deaf person; (3) communication disruption or who are unable 
to talk; (4) Tunagrahita or disturbance of intelligence;(5) Tunadaksa or physical disorders and health; (6) Barrel 
defect or disturbance of emotions and conduct;(7) learning difficulties, Slow learning; (8) the autistic; (9) Motor 
Disorders; (10) the victims of drug abuse, and (11) the combination of two or more types of above (Direktorat 
Jendral Pendidikan Dasar dan Menengah, Depdiknas, 2004, p. v.) 

III. METHODOLOGY 

In this research the special objective to be achieved is: denote validation of expertise (specialists in the 
field of Education Policy, Environmental education, environment and psychology) and empirical validation on a 
border group in school Inclusion in DKI Jakarta by teachers and students who are related, as the trial about the 
readability of the Guide, so that the result Guidelines are well prepared to be used in the primary Inclusion school 
(Special Services Education for social disabled children (underprivileged) in 2012 (Riant, 2004, p. 167). 

By using the methods which are created, namely: Studies in the field and the preparation of Documents in 
the form of a guide to Environmental education throughout Indonesia, Validation of expertise (an expert in the field 
of Education Policy, Environmental education, environment and psychology) and empirical validation on a limited 
group in inclusion school in DKI Jakarta by teachers and students who are related, as thetrial about the readability 
of the Guide, so that the resulting Guidelines are ready to be used in the primary Inclusion (Special Services 
Educationfor social disabled children(underprivileged) in 2012. 

This activity was conduct during a period of 3 months, September, October and November 
2011. In December 2011, the last Report was sent to the Directorate for special education and Special Services, 
Directorate of primary education. 
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IV. 


THE RESULTS OF THE STUDY 


Seeing the inference of international, regional and national policy above, Faculty of social sciences put an 
important role to carry out the Tri Darma of University to obtain input through academic studies or research results 
and workshop to collect a guide about education PK and LK for Environmental education. Throughout Indonesia, 
as a guide for principals and teachers in providing Special Services for groups of children aged 6-15 years emerged 
from marginalized society. 

Based on the phases inrealizing the fund assistance of subsidized program for College mentoring 
accountably and transparently in the preparation of guidelines for special education and special services for 
primary Environmental education. Through the varied steps of activities that has been carried out, namely: needs 
analysis, analysis of survey results, workshop and finalization, have resulted in: 9 (nine) PK LK Handbook on 
Environmental education. Throughout Indonesia, as a guide for principals and teachers in providing Special 
Services for groups of children aged 6-15 years came from marginalized societies. As for the content of the book is 
a guideline of grades 1 to 9. The Distribution of abilities and context of material as well as the assessment process 
and the results. 

The guidelines of PK and LK about Environmental education, this has contributed obviously to the 
achievement of the vision of the Ministry of education and national culture is to conduct a national educational 
services to make the Indonesian people who are smart and competitive. While the mission is: (1) availability: 
Increase the availability of services of education; (2) Affordability: affordability Expands education services; (3) 
quality: improving the quality and relevance of educational services; (4) Equality: Manifest equality in obtaining 
educational services; (5) the assurance: ensure the certainty of obtaining education service. The Directorate of 
special education and special services is an integral part of the effort and fundamentals of coaching and developing 
the national education. 

In addition, the guidelines of PK and LK about environmental education policy (PLH), agreed on 
February 19, 2004 by 4 (four) Departments, the Ministry of State for the environment (KNLH), Ministry of 
national education. Religion Ministries, and Ministry of Internal Affairs. 

V. RECOMMENDATION 

Based on the results of the study/research/surveys, workshops, discussion and conclusion then it can be put 
forward recommendations as follows: (1) Demand to the Central Government i.e. Ministries of culture and 
education, to keep going on doing socialization and fortification and revitalization of all Ministry informas 
regarding to be aware possibly about the policy of UNESCO, and the correlated National Policy with EE; (2) To 
headmasters and teachers of all subjects/ class teachers, in regular schools not only inclusion but also exceptional 
schools, as well as galleries for kids particularly because of marginal social field, to continue to maintain the 
mandate responsibility professionals respectively, so that the enhanced knowledge and skill in the performance of 
special education and special services with guidelines to the PK and LK rule about education for sustainable 
development (EfSD) as a result of this agenda. 
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Teachers’ Perceived Ability to Integrate Technology into the 

Instructional Setting 

Mabel CPO Okojie& Anthony A Olinzock 
Mississippi State University 


Abstract 

The purpose of this study was to assess teachers’ perceived expertise in using word processing, 
spreadsheet, and presentation software applications to facilitate instruction. The participants of this study were 313 
teachers who taught in the rural schools in the southern region of the U.S. The instrument for data collection was 
validated by professionals in the area of technology integration, and a Cronbach's alpha was used to establish the 
internal consistency of .95. The findings indicated that the participants’ responses to 30general statements related to 
their ability to provide instruction using word processing, spreadsheet, and presentation applications indicated that 
they did not express competence in over half of the questionnaire items. The conclusion is that the majority of 
teacher participants acknowledged that they lack competence to use these software applications to facilitate 
instruction. 

Introduction 

As new instructional technologies emerge, most educational institutionsstruggle to make these technologies 
available to their teachers. The researchers in this study observed that little or no effort is made 
byadministratorstoevaluate the technology expertiseof the teachers who will be responsible for integrating 
thenewtechnologiesintotheinstructional environment.Not only is it important that teachers have expertise in using 
these technologies, they must also be able to integrate them into the instructional setting so that they can foster 
meaningful learning. As technology advances with breathtaking speed, schools continually try to catch 
up.Byhavingaccess to a variety of social networks, most students use new technology on a daily basis as they 
integrate a variety of media through what they see as routine communications. 

Few schools evaluate teachers’ actual or perceived technology competence. In most cases it is assumed 
that teachers keep up to date through advanced education, training, and experience. Facilitating instruction using 
technology is a process that encompasses several steps. Its most basic steps include presenting the technology, 
practice using the technology, feedback from the teacher or others, and application of the technology in a realistic 
setting. However, teachers’ use of technology to support instmctionmay be hamperedif they lackthe competence to 
usetechnology themselves, includingthe knowledge and the ability to use appropriate pedagogical principles to 
implement technology integration (Okojie, 01inzock,&Boulder, 2006). 

Statement of the Problem 

Word processing, spreadsheet, and presentation software applicationshave become necessary skills in 
today’s teaching, learning, andresearch environments. Yet, do teachers have competence in using and integrating 
these technologies? For technology to be integrated successfully into teaching and learning, teachers not only need 
to understand the basic skills, but also need to understand how (methods/techniques) technology can be integrated 
into the curriculum (Baylor & Ritchie, 2002, Becker, 2001; Roberts, 2003). According to Becker (2001), “... the 
ways that teachers have their students use computers are certainly affected by their own level of technical expertise” 
(p.4). This study was designed to assess teachers’ perceived skills in using word processing, spreadsheet, and 
presentation software applicationsas integrated teaching tools. The findings can be used to assist individuals in 
developing appropriate instructionalmaterials that could addressteachers’ needs and weaknesses.A secondary 
purpose of this study was to determine if differences exist among teachers in their perceived skill in implementing 
technology integration based upon gender and age. 
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Research Questions 


The following questions guided this study 

1 . How do teachers evaluate theirability to facilitate instruction using word processing, presentation, and 
spreadsheet software applications? 

2. Do statistical differences exist among teachers in their perceived ability to facilitate instruction usingword 
processing, presentation, and spreadsheet software to facilitate instruction based upon gender? 

3. Do statistical differences exist among teachers in their perceived ability to facilitate instruction using word 
processing, presentation, and spreadsheet software to facilitate instruction based upon age? 

Educational Significance 

In order to provide appropriate technology training forteachers, technology trainers must know how these 
individuals (teachers) perceive their skill competence and their ability to integratesoftware applications into the 
teaching and learning environment. Quite often trainers assume that they know what teachers ortraineesneed as they 
prepare materials. Understanding teachers’ needs through research initiatives will provide opportunities for trainers 
to diagnose teachers’ needs and to provide appropriate instruction during professional development training. The 
researchersbelieve that teachers will learn more effectively if they are involved in their training because, as 
Knowles, Elwood, and Swanson (1998)state that “. . .adults resent and resist situations in which they feel others are 
imposing their wills on them” (p. 65). 


Review of Related Literature 

Computer technology has permeated all aspects of our economic and social lives, including educational 
practices. Educational institutions continued to strive to make technology the center piece of instructional tools. The 
challenge to integrate technology into the classroomcontinues to be an on-going concern. Since teachers direct 
classroom activities, teacher education, too, must address the integration of technology in the learning 
environment.Many school systems have invested large sums of money to provide technology training sessions and 
workshops for teachers. Woodbridge (2004) explained that computer technologies have been considered educational 
tools for over 40 years. During these years, public school systems and state and federal governments have spent 
billions of dollars to integrate technology into teaching and learning (Franklin, Turner, Kariuki,& Duran2001). 
Zuniga (2010)pointed out that over $2.8 billion has been spent on technology infusion from 2002 to 2008. Zuniga 
also pointed out that teachers have the desire to use technology to facilitate instruction, but lack of experience and 
lack of knowledge of technology integration inhibited their desire to do so. 

Discussing some of the views expressed by teachers on technology infusion, Zuniga (2010) stated that out 
of the 30 teachers who participated in his qualitative research, nine (30%) rated their computer skills between one 
and three on a ten-point scale, with one being the lowest and ten the highest. Zuniga reported that a rating in this 
range indicated that these teachers were hesitant in using computers for instruction. Eight out of the 30 participants 
rated their computer knowledge between four and seven, a rating that indicated moderate use; while 13 rated their 
knowledge between eight and ten which showed that they considered themselves knowledgeable in using computers. 
According to Zuniga, one teacher who participated in the study indicated that her school district explained the 
importance of computers in education and encouraged teachers to use computers for instruction. However,the 
teacher maintained that there was no evidence that computers were being used in the classrooms. Zuniga also 
reported that another teacher admitted that computer resources were available for use in the classrooms, but she did 
not use them because she did not know how to use computers. The views expressed by these teachers showed that 
the availability or lack of availability of computer technologies in the classrooms was not the only obstacle to 
technology integration. Teachers need regular training and frequent encouragement and reinforcement if technology 
integration is to be successful. The availability of classroom computers, alone, does not ensure that technology 
integration will take place. 
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Research Methodology 


Research Design and the Participants 

The design of this study was a survey questionnaire. Survey design is a self-reporting method of data 
collection. Gay, Mills and Airasian (2012) acknowledged that a survey represents one of the ways used to collect 
information, or data, on people’s opinions to describe certain characteristics of a given population. Seven schoolsin 
four counties were involved in this study; they were located in one of the southern states of the United States. The 
researchers received permission from the superintendents of each school district as well as the principals of each 
school involved in the study for teachers to complete the questionnaire. Teachers who participated in the study 
volunteered to do so. Most schools participating in the study were located in the rural areas. Atotal of 589teachers 
from various types of schools completed the survey; however, 313 (53%)surveys were complete and usable as 
shown in Table 1. 

Table 1 


Distribution of Teachers by Type of School and Percentages 


Types of Schools 

Number of Teachers 

Percentages 

Elementary Schools 

134 

42.8 

Middle Schools 

85 

27.2 

High Schools 

85 

27.2 

Career and Technical Schools 

4 

1.3 

Missing Data 

5 

1.6 


Instrumentation 

A survey questionnairewas used to collect data for this study. The questionnaire was reviewed and 
validated by two (2) professionals in the field of education who have experience in using technology to facilitate 
instruction. The purpose of the review was to make sure that the questionnaire itemswere meaningful and clear.The 
questionnaire was divided into two sections. Section Awasused to gather demographic information, and section 
Bwas used to collect data on teachers’ perceived ability tointegrate softwareapplications into teaching and learning. 
The questionnaire used for this study contained 30 items, and it was based on a 5-point Likert scale. The 
questionnaire was completed by 27 teachers in a pilot study on two occasions to establish the internal consistency of 
the questionnaire; those teachers did not participate inthe actual study. The internal consistency of the questionnaire 
itemsfor the first administrationwas.71. The researchers observed that several questionnaire items received 
inconsistent ratings. As a result, the researchers revised and/or replaced those questionnaire items. The questionnaire 
was administered to the same teachers again, and the Cronbach’s alpha result was .95. 

Data Analysis Procedure 

Data was analyzed using mean scores, percentages, t-tests, andANOVAs, including Turkey HSD Post hoc 
tests. The t-Test was used to determine if statistical significant differences existed between female and male 
participants in their responses to questionnaire items. The ANOVA was used to identify differences among groups 
of the participants based on eight age groups. HSD Post hoc tests were used to determine where difference existed 
among the different age groups represented in the study. 

Data Analysis 


Participants’ Demographic Data 

Table 2 shows the distribution ofparticipants by gender, ethnicity, and age. The majority of the participants 
were female, 221 (70.60%) and Caucasians 221 (70.60%). The age of the participants ranged from 20 years to 56 
years and above. 
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Table 2 

Dis tribution Participants’ Demographic Data by Frequencies and Percenta ges 


Demographic Variables 

Frequencies 

Percentages 

Gender 

Female 

221 

70.60 

Male 

91 

29.07 

Missing Data 

1 

.31 

Ethnicity 

Caucasians 

221 

70.60 

African American 

73 

23.32 

Hispanic American 

1 

.31 

Asian Americans 

2 

.63 

Missing Data 

16 

5.11 

Age 

20-25 

31 

9.90 

26-30 

3 

.95 

31-35 

35 

11.02 

36-40 

52 

16.06 

41-45 

29 

9.03 

46.50 

30 

9.06 

51-55 

48 

15.03 

56 and above 

54 

17.02 

Missing data 

31 

9.90 


Research Question 1 

Research Question 1 was: How do teachers evaluate their ability to facilitate instruction using word 
processing, presentation, and spreadsheet software applications? Participants evaluated their skill on a 1 to 5 scale 
with 1 representing strongly disagree and 5 representing strongly agree. The results indicated that the participants did 
not perceive that they possessed the skill toteach more than half of the items listed on the survey; they reported 
Undecided or Disagree ratings for 16of the 30 items as shown in Table 3. They did not strongly disagree or strongly 
agree on any item on the instrument. The highest mean score, 4.29, was for Question 1: 1 possess the skill to teach 
students how to create documents using word processing software. The lowest mean score, 1.90, was for 
Questionnaire item 2: 1 possess the skill to teach students how to format documentsusingword processing software. 

Table 3 


Participants Mean Scores for Survey Items Used to Facilitate Instruction 


Questionnaire Items 

Word Processing Software 

N 

Mean 

Scores 

SD 

1 -1 possess the skill to teach students how to create documents using word processing 
software. 

311 

4.29 

.950 

2 -I possess the skill to teach students how to format documentsusing word processing 
software. 

312 

1.90 

1.081 

3 -1 possess the skillto teach students how to edit documentsusing word processing 
software. 

313 

4.20 

.981 

4 -1 possess the skill to teach students how to merge documents using word processing 
software. 

312 

3.69 

1.205 

5 -I possess the skill to teach students how to edit documents using word processing 
software. 

308 

3.89 

1.232 

6 -1 possess the skill to teach students how to insert symbols using word processing 
software. 

310 

3.82 

1.232 

7 -I possess the skill to teach students how to attach documents to 
e-mail. 

308 

3.96 

1.269 


4 


524 



8 -I possess the skill to teach students how to use tracking tools to make changes using 
word processing software. 

9 -I possess the skill to teach students how to use drawing tools using word processing 
software. 

10 - 1 possess the skill to teach students how to organize documents using word 
processing software. 


310 

3.29 

1.248 

310 

3.35 

1.332 

311 

3.78 

1.138 


Presentation Software 

1 1 -I possess the skill to teach students how to 

12- 1 possess the skill to teach students how to 
the instructional goals. 

13- 1 possess the skill to teach students how to 
presentation software. 

14 -I possess the skill to teach students how to 
software. 

15- 1 possess the skill to teach students how to 
presentation software. 

16- 1 possess the skill to teach students how to 

17- 1 possess the skill to teach students how to 

18- 1 possess the skill to teach students how to 

19- 1 possess the skill to teach students how to 

software. 

20 -I possess the skill to teach students how to 
area. 

Spreadsheet Software 

21-1 possess the skill to teach students how to 

22 -I possess the skill to teach students how to 

23 -I possess the skill to teach students how to 
software. 

24 -I possess the skill to teach students how to 

25 -1 possess the skill to teach students how to 
software. 

26 -I possess the skill to teach students how to 

27 -1 possess the skill to teach students how to 


use presentation software. 

select appropriate design templates based on 

change the size of images inserted into 

change slide designs using presentation 

apply slide transition effect to slides using 

insert sound using presentation software, 
insert animation using presentation software, 
insert charts using presentation software, 
change the order of slides using presentation 


format a chart by changing color of the plotted 


input data using spreadsheet software. 

modify data on a spreadsheet. 

use data to create pie charts using spreadsheet 

insert rows, columns and cells on a spreadsheet, 
perform arithmetic functions using spreadsheet 

copy a range of cells on a spreadsheet, 
delete rows and columns on a spreadsheet. 


28 -1 possess the skill to teach students how to 

29 -1 possess the skill to teach students how to 
software. 

30-1 possess the skill to teach students how to 
calculations on a spreadsheet. 


sort a list on a spreadsheet. 

use various functions using spreadsheet 

use formulas to perform various mathematical 


311 

3.72 

1.217 

311 

3.61 

1.221 

310 

3.55 

1.325 

308 

3.56 

1.334 

310 

3.59 

1.263 

307 

3.38 

1.367 

308 

3.37 

1.348 

309 

3.30 

1.280 

309 

3.64 

1.268 

306 

3.09 

1.379 

308 

3.35 

1.341 

309 

3.31 

1.332 

308 

3.07 

1.365 

308 

3.48 

1.352 

309 

3.08 

1.315 

308 

3.36 

1.351 

309 

3.51 

1.340 

308 

3.17 

1.365 

306 

3.09 

1.379 

308 

3.35 

1.341 


Research Question 2 

Research question 2 was: Do significant differences exist among teachers in their perceived ability to use 
softwareapplication tools to facilitate instruction based on gender? A confidence level of .05 (p. < .05) was set a 
priori for all statistical tests. In the area of word processing, female respondents had significantly higher scores for 
questionnaire items 1, 3, 5, 6 and 7 than male participants as shown in Table 4. For questionnaire item 2, male 
participants had a significantly higher score than female participants. No significant differences were found for any 
of the presentation or spreadsheet skills listed on the questionnaire based on gender. 
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Table 4 


T-Tests for Word Processing Questionnaire Items based on Gender 


Survey Item 

Mean - Male 

Mean - Female 

t 

P- 

Q1 —I possess the skill to teach students how to create 
documents using word processing software. 

4.02 

4.41 

2.88 

.01* 

Q2 -I possess the skill to teach students how to format 
documents using word processing software. 

2.11 

1.81 

2.07 

.05* 

Q3 -I possess the skill to teach students how to edit 
documentsusing word processing software. 

3.98 

4.29 

2.37 

.05* 

Q5 -I possess the skill to teach students how to edit 
documents using word processing software. 

3.36 

4.00 

2.36 

.05* 

Q6 - 1 possess the skill to teach students how to insert 
symbols using word processing software. 

3.56 

3.94 

2.36 

.05* 

Q7 -I possess the skill to teach how to attach documents 
to e-mail. 

3.59 

4.11 

3.03 

.01* 


*p. < .05 


Research Question 3 

Research Question 3 was: Do significant differences exist among teachers in their perceived skills to use 
word processing, spreadsheet, and presentation software to facilitate instruction based on age?The participants were 
divided into eight groups based on age: 20-25, 26-30, 31-35, 36-40, 41-45, 46-50, 51-55, and 56 or higher. The 
researchers used an ANOVA to determine if significant differences existed for any questionnaire item based on age, 
and Tukey HSD Post hoc tests were used to determine where the differences existed. As shown in Table 5, with the 
exceptions of questionnaire items20 and 25, significant differenceswere found for allitemsat p. < .05. The 
participants were undecided for questionnaire items20, and 25. The three lower age groups 20-25, 26-30, and 31-35 
did not differ significantly from one another, and the three higher age groups 46-50, 51-55, and 56+ did not differ 
significantly from one another.For items on the questionnaire that showed a significant difference based on age, the 
threelowest age groups differed significantly from the three highest age groups, with the lower age groups indicating 
that they felt more competent in using software applications in teaching than older participants. 

Table 5 


ANOVA Tables for Questionnaire Items based on Age 


Word Processing 


Sum of Squares 

df Mean Square 

F 

Sig. 

Qi 

Between Groups 
Within Groups 

Total 

38.160 

229.059 

267.218 

7 

299 

306 

5.451 

.766 

7.116 

.000* 

Q2 

Between Groups 

46.490 

7 

6.641 

6.613 

.000* 


Within Groups 

301.302 

300 

1.004 




Total 

347.792 

307 




Q3 

Between Groups 

38.143 

7 

5.449 

6.682 

.000* 


Within Groups 

245.449 

301 

.815 




Total 

283.592 

308 




Q4 

Between Groups 

59.643 

7 

8.520 

6.711 

.000* 


Within Groups 

380.877 

300 

1.270 




Total 

440.519 

307 




Q5 

Between Groups 

32.183 

7 

4.598 

3.231 

.003* 


Within Groups 

421.238 

296 

1.423 




Total 

453.421 

303 
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Q6 

Between Groups 
Within Groups 

Total 

27.881 

429.272 

457.154 

7 

298 

305 

3.983 

1.441 

2.765 

.008* 

Q7 

Between Groups 

50.119 

7 

7.160 

4.954 

.000* 


Within Groups 

427.763 

296 

1.445 




Total 

477.882 

303 




Q8 

Between Groups 

23.886 

7 

3.412 

2.259 

.030* 


Within Groups 

450.052 

298 

1.510 




Total 

473.938 

305 




Q9 

Between Groups 

37.649 

7 

5.378 

3.179 

.003* 


Within Groups 

504.233 

298 

1.692 




Total 

541.882 

305 




Q10 

Between Groups 

41.245 

7 

5.892 

5.059 

.000* 


Within Groups 

348.234 

299 

1.165 




Total 

389.479 

306 





Presentation 


Qll 

Between Groups 
Within Groups 

Total 

77.190 

370.907 

448.098 

7 

299 

306 

11.027 

1.240 

8.889 

.000* 

Q12 

Between Groups 
Within Groups 

Total 

69.451 

382.471 

451.922 

7 

299 

306 

9.922 

1.279 

7.756 

.000* 

Q13 

Between Groups 
Within Groups 

Total 

42.519 

492.922 

535.441 

7 

298 

305 

6.074 

1.654 

3.672 

.001* 

Q14 

Between Groups 
Within Groups 

Total 

77.162 

459.387 

536.549 

7 

296 

303 

11.023 

1.552 

7.103 

.000* 

Q15 

Between Groups 
Within Groups 

Total 

76.532 

409.027 

485.559 

7 

298 

305 

10.933 

1.373 

7.965 

.000* 

Q16 

Between Groups 
Within Groups 

Total 

67.773 

496.273 

564.046 

7 

295 

302 

9.682 

1.682 

5.755 

.000* 

Q17 

Between Groups 
Within Groups 

Total 

67.964 

481.773 

549.737 

7 

296 

303 

9.709 

1.628 

5.965 

.000* 

Q18 

Between Groups 
Within Groups 

Total 

30.443 

466.586 

497.030 

7 

297 

304 

4.349 

1.571 

2.768 

.008* 

Q19 

Between Groups 
Within Groups 

Total 

88.785 

395.753 

484.538 

7 

297 

304 

12.684 

1.333 

9.519 

.000* 
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Q20 

Between Groups 
Within Groups 

Total 

16.463 

557.355 

573.818 

7 

294 

301 

2.352 

1.896 

1.241 

.280 

Spreadsheet 

Q21 

Between Groups 

47.450 

7 

6.779 

4.037 

.000* 


Within Groups 

497.020 

296 

1.679 




Total 

544.470 

303 




Q22 

Between Groups 

44.124 

7 

6.303 

3.787 

.001* 


Within Groups 

494.388 

297 

1.665 




Total 

538.511 

304 




Q23 

Between Groups 

33.766 

7 

4.824 

2.682 

.010* 


Within Groups 

532.339 

296 

1.798 




Total 

566.105 

303 




Q24 

Between Groups 

42.066 

7 

6.009 

3.488 

.001* 


Within Groups 

509.905 

296 

1.723 




Total 

551.970 

303 




Q25 

Between Groups 

14.292 

7 

2.042 

1.183 

.312 


Within Groups 

512.659 

297 

1.726 




Total 

526.951 

304 




Q26 

Between Groups 

47.384 

7 

6.769 

3.965 

.000* 


Within Groups 

505.353 

296 

1.707 




Total 

552.737 

303 




Q27 

Between Groups 

65.066 

7 

9.295 

5.763 

.000* 


Within Groups 

479.046 

297 

1.613 




Total 

544.111 

304 




Q28 

Between Groups 

58.878 

7 

8.411 

4.919 

.000* 


Within Groups 

506.171 

296 

1.710 




Total 

565.049 

303 




Q29 

Between Groups 

29.339 

7 

4.191 

2.567 

.014* 


Within Groups 

486.608 

298 

1.633 




Total 

515.948 

305 




Q30 

Between Groups 

43.051 

7 

6.150 

3.703 

.001* 


Within Groups 

484.949 

292 

1.661 




Total 

528.000 

299 





*p. < .05 


Discussion of the Findings and Conclusion 

Teachers’ responses to 30 general software statements related to their ability to teach students to use word 
processing, presentation, and spreadsheet applications indicated that they did not express competence in over half of 
the items. They agreed (a mean of 3.5 or higher) that they felt competent in 14 of the 30 questions, 7 in the word 
processing area, 6 in the presentation area, and only 1 in the spreadsheet area. This finding supports Littrell, 
Zagumny,and Zagumny (2005) who believe that teachers do not yet possess the skill to implement technology 
integration. Keengwe, Onchwari,and Wachira (2008) reported that teachers do not have confidence in using basic 
software applications in teaching. Theseauthors concluded that computer integration has not been properly and 
effectively infused into the curriculum activities. Eteokleous(2008) maintained that some of the teachers who took 
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part in her study indicated that they use technology, but remarked that such use does not include any innovative 
applications. Female participants reported significantly higher levels of competence in five of the ten areas of word 
processing than males. Flowever, there were no differences based on gender for any questionnaire items in the 
presentation and spreadsheet applications... 


Recommendations 

It is recommended that training which involves peer-to-peer learning, collaborative activities, a variety of 
training materials, and a skilled trainer can provide a learning environment in which individual differences can be 
accommodated. It is important to remember that training is only one step in the learning process, it is equally 
important that follow-up training activities in the form of workshops be instituted in order to help teachers 
consolidate the skill they have acquired. A potential technology trainer can send a diagnostic questionnaire to 
teachers prior to the training session aimed at determining their (teachers) strengths and weaknesses so that these 
weaknesses can be addressed during the training sessions.lt is recommended that the training materials be based on 
the trainees’ (teachers) needs and weaknesses and not solely determined by the trainer. 
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Abstract 

This study aims to design and develop a feasible instructional module that could enhance grad students’ 
correcting exposure abilities. The instructional module was developed based on situated theories. The instructional 
module includes the online instructional module and the online photo gallery and social network. The online 
instructional module provides the students meaningful and authentic problems and the online photo gallery and 
social network provides a home for online community of practice and authentic assessment. 

Introduction 

Thanks to advancement in the digital camera market, people can find the camera that serves their needs 
best. One of the advantages of digital cameras today is that they have auto mode which is basically camera is taking 
control of making critical decisions about exposure and image quality. But even the best camera with auto mode is 
far away from knowing what people really want to shoot. Let’s assume that there is a scene with beautiful panorama 
in the background and a human subject as a model. The camera cannot decide whether the background should be 
blurred or sharpen in other words the camera cannot decide on its own about the image in people’s mind. If anyone 
use auto mode without understanding exposure, he or she can end up with the image that is different from the image 
in his or her mind. To shoot the image that is intended, people should have the control of their camera’s decision- 
making process and they can do it by understanding exposure. 

Exposure can be defined as the amount of light that falls on each unit area of an image sensor. In other 
words it is controlling the amount of light that enters to an image sensor in a camera. Camera has mainly three 
mechanisms for controlling light therefore exposure; shutter speed, aperture, ISO. The instructional module based on 
situated learning was designed and developed for Instructional Design and Technology (IDT) graduate students at 
Virginia Tech. After the completion of this module, IDT graduate students at Virginia Tech will be able to correct 
exposure by adjusting shutter speed, aperture size and ISO. And because the course was designed based on situated 
cognition, the students will be expected to use photography terminology related with exposure and participate fully 
in the community of practice. 


Needs and Audience Analysis 

There is no doubt that, IDT graduate students at Virginia Tech will be faced with shooting or selecting 
images for instructional materials they are developing as professional instructional designers after they will 
graduate. And before they graduate, as IDT graduate students, they again need to shoot images for their instructional 
materials or presentations. Therefore as instructional designer professionals or students, they should have some 
knowledge about shooting images that is above the knowledge of an end user. This course could help them to 
override camera’s decision-making process and to become a prosumer rather than a consumer. 
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The target audience is constituted of people who have different culture, professional and education 
backgrounds. They are all computer literate and have different online learning experience. Survey was conducted to 
investigate the learners’ current situation in regards to adjusting exposure. It was an online survey and the URL was 
posted to the Virginia Tech IT LIST server in which Virginia Tech IDT students are automatically subscribed to. A 
total of 30 students answered the survey. After analyzing the survey results, it is clearly seen that the learners are 
familiar with using digital compact cameras but have no or limited comfort in regards to adjust exposure. Most of 
them have taken a photo before for their class projects and again most of them think that they will take a photo for 
their academic and professional career. Most of the learners feel uncomfortable in using a camera in manual setting. 
They indicated that they mostly use auto controls and have no or little experience in camera setting. After analyzing 
the survey results, it is clearly seen that there is a need for instruction about adjusting exposure. 

Situated Learning 

Many researchers claimed that there is a separation between knowing and doing in traditional education 
(Brown, Collins & Duguid, 1989; Harrington & Oliver, 2000). This separation may lead students to perceive 
knowledge as an integral substance that is independent of situations in which it is learned and used and result in 
decontextualized or inert knowledge. Brown, Collins and Duguid (1989) stated that what is being learned cannot be 
separated from the context in which it is learned or applied. Situated learning is one of the important learning 
theories that emphasize and promotes authentic learning. In situated learning, the context that students learn skills 
and knowledge reflects how that knowledge or skills would be useful in real life (Collins, 1988; Leve & Wenger, 
1991). Therefore, situated learning can be used to provide students authentic learning environments and solve the 
inert knowledge problem. 

Situated learning provides students authentic learning environments that students can participate in the 
actual experience rather than being external to the event. Learners in the authentic learning environments interact 
with the values, norms, and true culture of a specific community or organization (Brown et al., 1989). Situated 
learning also provides learners chance to reflection to enable abstractions to be formed (Brown et ah, 1989). 

Communities of practice play vital role in situated learning. Lave and Wenger (1991) stated that in 
legitimate peripheral participation, learners become participants in communities of practice and move up in the 
community starting as a new entrant all the way to mastery which is in essence frill participation. As they move up 
in the community, learners leam skills, acquire knowledge and understand the artifacts and identities of the 
community. Driscoll (2000) stated that that it is a matter of time and experience within the community before a 
newcomer is allowed or given full access to the community’s resources. Brown et ah (1989) stated that peripheral 
participation is very important for those entering into a new culture. The newcomers have the opportunity to observe 
how old timers or practitioners in the community’s various levels interact and behave, for which they will 
understand the rules and culture of the community. 

Wenger (as cited in Driscoll, 2000) defined five types of learning trajectories; peripheral, inbound, insider, 
boundary, and outbound. Peripheral trajectory is used to define learners who never engage in full participation, 
inbound trajectory, on the other hand, is used to define learners who is invested in the community of practice and 
expected to fully participate. Insider trajectory is used to define members who continually evolve in the community 
of practice and members who draw insider trajectory are people who are also member of other related communities 
and outbound trajectory is sued to define members who are leaving the community. 

Situated learning also promotes cognitive apprenticeship and communities of practice support cognitive 
apprenticeship because new comers have chance to participate in the community by interacting with old timers and 
they gradually gain skills and knowledge and become old timers as time passes. 

Herrington and Oliver (2000) provided nine learning design elements to operationalize situated learning 
theory; (1) provide authentic contexts that reflect the way the knowledge will be used in real life, (2) provide 
authentic activities (3) provide access to expert performances and the modeling of processes (4) provide multiple 
roles and perspectives, (5) support collaborative construction of knowledge (6) promote reflection to enable 
abstractions to be formed, (7) promote articulation to enable tacit knowledge to be made explicit, (8) provide 
coaching and scaffolding by the teacher at critical times, and (9) provide for authentic assessment of learning within 
the tasks. 
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The Instructional Module 


Online Instructional Module 

The online instructional module will start with an introduction page (Figure 1). The introduction page will 
include a photo collage and a video. In the video there will be an agent who is wearing a Virginia Tech IDT t-shirt. 
She will say: 

“Hi lam Zeynep. Iam a PhD student in Instructional Design and Technology Program. I have a 
problem. I need to take many pictures to create educational materials for my course projects. 

Even though I always use programmed or auto option on my camera, the images are under or 
over exposed as you can see on this page. This is not only problem I have with programmed or 
auto option. For example, when I am shooting a fast-moving object and I want that object be 
blurred in my picture or when Iam shooting a portrait and I want the background be sharp, I 
rarely get the results I want. So, I have decided to override my camera ’s decision-making process, 
to get the image I want. One of my professors told me that the key to know how to take control of 
my camera is an understanding of exposure. I will need your help to understand the exposure. If 
you ready to start to help me, you can click on the Next button to continue. ” 
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Figure 1 Introduction Page (presenting a real problem, presenting a community) 

One of the aims of the introduction page is presenting students a real problem, which the learners could 
also have in their lives and asking for help to include them in a learning community. After the introduction page, the 
learners will be expected to take responsibility and help the agent to solve the problem. On this page and on the 
following pages, the agent will intentionally use photography tenninology. The aim here is making the learners get 
familiar with the terminology that they are expected to use in commercial photo gallery and social network and 
make the leaners start to use language of the community of practice. 

Cameras have mainly three mechanisms to control exposure; shutter speed, aperture and ISO. These three 
mechanisms has also individual functions such as shutter speed controls how long the film will be exposed to a light 
and it changes the way movement appears on the image, aperture size controls how much light will enter to an 
image sensor and it defines depth of the field, and finally ISO controls film’s sensitivity to a light and it defines how 
nosier the image will be. Therefore, each mechanism will have its own page and on these individual pages, the agent 
will present a real problem with an exposure but specifically related with that mechanism. On every page, there will 
be a link to the introduction to photography course of which the exposure module is part and a link to this course 
twitter account. The twitter account will be used as means to enhance community of practice. 

On the aperture page (Figure 2), there will be the agent video and a camera simulation. The agent will 
present a problem related with an exposure and aperture size. She will say: 

“Hi, I shot the image with these setting. You can click on the Snap Photo button to see the image. 

Actually I wanted have a blurred out background and neither too bright nor too dark image. You 
can click on return to viewfinder button to help me to adjust the exposure. If you need help you 
can click on small i button on the right upper corner. 
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The learners first should click on “Snap photo!” button to see the image she shot. And then they should 
correct the exposure and also make sure that the background is blurred out. They can correct the exposure by 
changing ISO or shutter speed but the aim of this page is making them to change aperture size and see its effect on 
the image. 
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Figure 2 Aperture page (authentic task) 



The simulation will look like a real camera’s viewfinder. The users will be able to see the scene they will 
shoot image of and they will be able to make changes in: aperture size, shutter speed, ISO, lighting, distance and 
focal length. The users will also be able to choose whether they will use a tripod or not and work with aperture 
priority, shutter priority and manual options. If they will need help, they will be able to click on small i button and 
get short information about the camera’s controls in the simulation (Figure 3). After they will click on “Snap photo!” 
button, they will be able to see the real image. 



Figure 3 Help mode of the simulation 

Situated cognition promotes authentic learning environments to solve decontextualized knowledge 
problem. And it claims that the learners should learn knowledge or skill in a context that reflects the way the 
knowledge woidd be useful in real life. Therefore, the aim of using a simulation in this module is provide the 
learners an authentic environment and activities that have real world relevance. The viewfinder in the simulation 
includes very basic mechanism that the learners can find in every camera. 

The learners will be able study on this screen as long as they want to study. They will be encouraged to 
take notes while they are playing with the simulation. There will be a text under the agent saying “Don't forget to 
take notes, while you are trying to correct the exposure!” After they will click on Next button to continue, they will 
see the agent and a text box on the next page (Figure 4). The agent will say: 

"Thank you for your help. Could you please write how you corrected my picture settings to get the 
correct image, in this box? I also asked the same question to my professor, after you submit your 
method, you will be able to see her answer. You can click on the continue button to continue. ” 
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The agent will ask them to reflect on what they did to solve the problem. The learners could come to this 
page without understanding the function of aperture and its effect on exposure. So this page has an aim to make 
them think about the solution. The text below the agent will remind them that they can look at their notes that they 
wrote while they are playing with the simulation. After they will write how they solved the problem, they will be 
able to see the how the expert solved the same problem (Figure 5). The aim of this activity is to promote reflection 
and provide access to expert performances and thinking and the modeling process. 



Figure 4 Reflection Page (promoting reflection) Figure 5 My Professor’s Method (providing access 

to expert performance) 

There will be the same flow for the shutter speed and ISO pages. The learners will first need to solve a real 
problem by changing related control mechanism on the simulation and then they will need to write how they solved 
the problem and they will have access to the expert performance. 

There will be another page on which the learners will need to correct exposure by controlling more than 
two of the mechanisms. There will be more than 4 problems on this page. And every time the learners will need to 
write how they solved the problem and will be able to see the expert’s method to solve the same problem. 

The online module will also have resources, dictionary and IX pages. On the resources page, there will be 
links to useful resources like photography courses on Lynda.com which VT students have free access, video 
tutorials, web tutorials, books and web pages of known photographers (Figure 6). The aim here is providing the 
learners more resources they coidd use after the instruction. On this page, students are not only provided with 
important and useful resources but they are also given a chance to suggest a new resource. The students will need to 
click on “Suggest a resource” link and write an e-mail saying what resource they like to add on this page and why 
they think it is important to have that resource. The students will need to write more than “because it is cool” or 
“because I like it”. They will need to write specifically what they liked about that resource. Aim of this suggestion is 
making them take responsibility for their friends in other words for their learning community. 
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Figure 6 Resource page (providing students more resources and chance to 
serve their community by suggesting a resource) 

This is also true for dictionary page. On the dictionary page, they will be able to see definition of terms that 
the agent is using and terms belongs to general photography terminology such as backlight, noise, lens, and 
calibration. The aim here is again making them get familiar with the language the community use. They will also be 
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able to suggest a new term on this page. They will need to define the term plus why they think it is important to have 
that term on this page. They will be expected to take responsibility for their community. 
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Figure 7 Dictionary page (proving terminology belongs to the community of practice) 

Finally, there will be page that provides a link to commercial photo gallery and social network; lx.com. On 
all of these pages, there will be also the agent video that gives the learners information about the page. 

Online Photo Gallery and Social Network 

The online photo gallery and social network site will have two aims; it will provide a home for online 
community of practice and authentic assessment. The learners will need to have an account and add their classmates 
as a friend. And they will need to upload three pictures every week and give information about how they took these 
pictures. And they will need to write comments on their friends’ pictures as their friends’ pictures appear on their 
timeline. 



atMH v HHRH 

'iniGii-t? : . mnen 


Figure 8 lx page, a gate to the learners’ community of practice 

The online photo gallery and social network site includes tutorials to teach photography, tutorials to teach 
rules about how to communicate and write critiques on this site. It also includes discussion forums, and navigation 
links to pictures or photographers. The learners will be able to interact with their classmates as well as other amateur 
or professional photographers. The instructor of this course and two teaching assistant will be also member of these 
online community. The previous students of this course will also have chance to continue to upload pictures and 
make comments to each other as well as to new students. 

Throughout to the course, the learners will need to prepare a photography portfolio that includes their 
pictures they shot. The learners will have the chance to shoot the same composition again after they got feedback 
from other members of the community. According to Collins, Brown, and Newman (1989) there are six critical 
features of cognitive apprenticeships. They are; observation, coaching, scaffolding, modeling, fading and reflection. 
In instructional module and in the commercial site cognitive apprenticeships will be supported by reflection, fading 
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and modeling. The learners will have chance to reflect on their solution method, access and model the expert 
performance in the online module and in the commercial community of practice site, and fading is provided by 
designing sequence of the lesson as starting from a problems that the learners deal with only one mechanism and 
ending with problems that students deal with all three mechanisms to correct the exposure. The instructor and the 
teaching assistance will also provide coaching and scaffolding in the online community of practice site. 

Motivation 

The introduction page will be used to take attention of the learners. The video is used to take attention of 
the learners. There will be another student from the same department on the screen who is asking for their help. The 
problem that the agent mentions here is a problem, which the learners can relate with their lives. And because the 
learners would use the skills that the course is teaching in their professional lives, it is assumed that the course will 
have high relevance. The practices will be used to increase confidence of the students. The instruction will start with 
basic task (dealing with one mechanism at a time) and will continue with more complex tasks (dealing with more 
than one mechanism at a time). And after completing of one task, the students will continue to the next step with 
more confidence. And because the learners will be dealing with an authentic task (simulation), it will maintain their 
attention. The learners will be able to take notes while they are playing with the simulation and they can use these 
notes while they are writing their method to solve the problem. This note taking would serve as a metacognition 
strategy. The online community of practice site would be also take attention of the learners because it is very similar 
to one of the currently most popular social network site. The site will provide notifications when one of the learners 
upload a new image or write a new comment. They will know if there is something new on the site. The notification 
could also help maintain the learners’ attention. And as they progress in the community they are expected to be more 
confident. Making comments to others, helping them to correct their images, working with the experts and getting 
feedback from the experts would increase their confidence. 

Assessment 

Driscoll (2000) stated that assessment in situated learning includes portfolios, summary statistics and 
diagnosis. In this lesson, portfolios will be the learners’ electronic photography portfolios that they developed over 
time and reflect the processes of learning. Summary statistics will be the user log information that commercial site 
provides. User logs will provide information basically about how long users stayed online on the site, how many 
images they uploaded, how many critiques they wrote for their friends etc. Diagnosis will be the instructor’s and 
teaching assistants’ continuous assessment of the learner’s progress and capabilities. Online discussion rubric and e- 
portfolio rubric will be used for grading. 

Formative Evaluation 

Toward the end of the design, field trial formative evaluation could be conducted with 20 IDT graduate 
students. An online survey could be conducted to investigate the learners’ current knowledge about exposure. And 
the students would be presented with lesson materials and given a month to participate in the community of practice. 
Assessment can provide information about effectiveness of the lesson. If students would have high points in rubric, 
it could be assumed that the lesson was designed to be effective. Otherwise, the designer would need to revise need 
and learner analysis, objectives, assessment and learning task. Possible mistakes in the design could be; the topic 
would not have high relevance or the learners would not have enough motivation to learn exposure, objectives 
would not reflect the nature of community of practice, assessment would not be the most appropriate assessment 
method for the target audience because there is diversity in terms of culture, background and experiences with 
online instruction, and finally learning activities would not be appropriate for the target audience or learning 
activities would not match with the theory as it is suggested. 
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Abstract 

With the onset of an aging baby boomer population and an increasing need for health care, the projected 
shortage of nurses continues to grow. Training registered nurses is paramount to care for these individuals 
(American Association of Colleges of Nursing, 2013). High-fidelity simulation (HFS) is one of the methods to help 
educate students without harm to patients. This technique is becoming more prevalent in nursing education 
providing the opportunities to train students using various patient care scenarios (Jeffries, 2009; Nehring & Lashley, 
2010). First year pre-licensure baccalaureate nursing students are given the opportunities to repeat procedures under 
different patient care scenarios using high-fidelity simulations (HFS). Data was collected from 50 students during 
eight different medical conditions during a complete semester. Variables dealing with patient safety issues and other 
medically appropriate variables were recorded depending on the procedures needed for the medical condition. This 
echoing gave students the opportunity to practice skills needed in their profession under varying situations. 

Several issues draw attention to the need to use simulation to instinct basic skills to nursing students. A 
major issue suggested by (Del Bueno, 2001; Feingold, Calaluce & Kallen, 2004) pointed to the lack of access to 
clinical placements to develop these basic skills. Another issue stated by Rystedt and Lindstrom (2001) was an issue 
of patient safety of patients practicing on actual patients. Agency for Healthcare Research and Quality (AHRQ) 
(2003) explained patient safety as “Freedom from accidental injury or avoiding injuries or harm to patients from 
care that is intended to help them” (p.8). The limited experiences by students in clinical placements can be a 
drawback due to a lack of standardized learning opportunities in a clinical setting (Medley & Horne, 2005). Jeffries 
(2005) proposed that the acquisition of basic skills such as patient-centered care could be presented effectively with 
simulations as an efficient choice versus clinical placements. Student engagement theory has been suggested as the 
framework of using simulations as a teaching methodology in the nursing by Jefferies (2005, 2007) 

Suggesting that individuals entering the nursing field presently were introduced to using computer simulation in 
other venues, today’s High Fidelity Simulation would improve student knowledge retention and critical-thinking 
skills (Sportsman, Puetz & Billings, 2008). Yee et al. (2005) working with anesthesia residents presented a report 
which suggested that a single experience with HFS would improve the ability of the students cognitive and 
interpersonal skills. In addition different clinical scenarios allow student to practice procedures without putting 
patients in harms way (Issenberg, McGaghie, Petmsa, Gordon & Scalese, 2005). For nursing the use of simulators 
provides an opportunity to give accurate and safe care practice to students before their experiences in hospital 
environments (Sanford, 2010). 

The definition of simulation in health care was suggested (Schiaventato, 2009) as “techniques used to 
represent nursing processes and actions in an educational context.” High-fidelity simulations in nursing involve a 
mannequin with the ability to present different scenarios echoing medical conditions. In nursing education, it presents a 
realistic medium to acquire basic procedures and patient care (Gaba, Howard, Fish, Smith & Sowb, 2001; Shaffer & 
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Armstrong, 2001). These mannequins supplement a human patient by representing physiological and vocal responsive 
in real time. An example would be a Laerdal SimMan@ University Patient Simulator which presents a realistic weight 
distribution, joint articulation with lifelike sounds of the heart, lung and bowel, vocalization, realistic airway and 
pulmonary mechanics and pulses found that the carotid, femoral, brachial and radial locations. 

One purpose of simulated clinical experiences using High Fidelity Simulations (HFS) for educating pre- 
licensure baccalaureate nursing students is to explore the effect of echoing opportunities to perform proper nursing 
practice techniques. Previous design utilizing HFS gave students a four step process with the same scenario: first 
scenario attempt, instructor -led debriefing, second scenario attempt followed by a final debriefing. This former 
method required more time to introduce different medical procedures and the delay time between repeats was too 
short to allow students to integrate the proper practices. New design allowed fifty nursing students exposure to eight 
different patient care scenarios using a HFS to prepare them for competent patient care. The patient condition 
presented during simulations were: l)Safe Entry, 2)Isolation Precautions, 3)Safe Entry and Vital Signs, 4)Safe Entry 
and Medication Administration, 5)Hyperglycemia, 6)End-of-Life and Pain Assessment, 7)Pediatric Injury, and 
8)Fluid Imbalance. Basic safety procedures were expected during all eight scenarios while some situation specific 
procedures were collected during only a few of the scenarios. Procedures which were recorded varied with the 
patient conditions allowing students to practice basic care and safety procedures under conditions that a new 
scenario presented simulated clinical experiences (SCE) are similar to what they would experience in a hospital 
setting. These variations allowed the investigation of whether the learning was remembered in a different care 
environment and were not context dependent. 


Research Hypothesis 

Did performance of basic care and safety’ procedures improve with repetitive participation in simulated clinical 
experiences according to student self-report? 

Methods 

The purpose of this study is to determine if repeating healthcare simulations by nursing students decreases errors or 
omissions. The faculty of the University of West Georgia created several simulation classrooms replicating a 
hospital setting. Control rooms were located adjacent to the simulation laboratory to allow instructors to observe the 
nursing students perform the scenario, and to adjust the mannequin’s reactions via computer control to what occurs 
during the simulation. This high fidelity simulation environment allows the student the ability to “do over” or repeat 
a simulation if necessary which would be impossible in a live clinical environment. The teaching model followed 
the simulation model presented by Jeffries (2005) and illustrated in Figure 1. 



Figure 1: Jeffries teaching model 
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Sample 


Convenience sample of fifty full-time pre-licensure baccalaureate nursing students in their 1 st clinical semester were 
selected to participate in a semester long class using high-fidelity HPS or SCE simulated situations. 

Instruments 



Figure 2: Gaumard® 5 year old simulator 

Hal® (see figure 2) is a five year old pediatric simulator with human like responses of breathing, circulation and 
color change, active eyes, venous access, blood pressure, etc. Activities are controlled by the instructor and software 
controlled scenarios. Touchscreen monitors show students vital signs reproducing a real clinical setting 



Figure 3: SimMan Classic 

SimMan Classic® ( see figure 3) from Laerdal Corporation® has the ability to present lifelike simulations 
of conditions of the head, neck, chest abdomen and extremities control by instructor and software scenarios 

Assessment 

Once students finished the ten to fifteen minute simulation, they were given a check list composed of 74 
data points related to scenario-specific desired nursing actions to capture what they believed they performed or did 
not perform, allowing them to reflect on what they did during the simulation. The questionnaire was based on the 
Creighton Competency Evaluation Instrument CCEI created from the original tool Creighton Simulation Evaluation 
Instrument and is the second generation (see Appendix A) (Todd & Manz, 2009). The observing faculty also filled 
out their copy of the same check list during the simulation. Then the students interacted with the faculty to watch 
and listen to feedback for each of the student’s actions via audiovisual playback in a classroom remote to the 
simulation lab. During this debriefing session, the faculty first reviewed what was done correctly, and then mistakes 
or omissions were discussed. After one or two weeks the students were then given the opportunity to repeat the 
simulation. The procedure followed the same format as the first attempt with each step being repeated 
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Setting 


SimMan® and Hal® allow for a protocol to be developed to reproduce the simulated experience of patient 
care through computer programming of a clinical scenario. A simulated hospital room with a nursing station was 
created. Medical records for the patient of the day were given to the students before the simulations. Both are 
outfitted with the proper tubes, drains, and dressings consistent with the clinical history. An observation room 
adjacent to the hospital room built with a one-way mirror was set up as a control center for the scripted audio 
responses and physical actions of the mannequin. Audiovisual equipment recorded each of student nurses’ responses 
and behaviors during the simulation which would be used later in the feedback stage. Students are then given the 
opportunity to work in small groups of two or three to enter the simulation room, assess patient needs and intervene 
appropriately. 


Methodology 

Student nurses are taught safety procedures for the protection of the client and protection of the nurse. 

These commonly include hand hygiene, checking the environment for abnormalities, correctly identifying the 
patient and ergonomic safeguards for the nurse. These would later be performed during the simulation experience. 
First simulation experience was performed in groups of two or three, followed with a self-reported questionnaire and 
a debriefing by observing faculty. No leader was chosen for the group with the students to perform in the role of a 
nurse. The echoing simulation was under taken after a delay of one to two weeks. The second simulation was 
followed with student filling out their second self-report on their performance. 

Data was transcribed into an Excel© spreadsheet and later put into SPSS 19© for in depth analysis. 

Initial Results 

This study looked at the results of two simulations: Asthma and Hypot Hypoglycemic. The results 
demonstrated a growth in student practice during these simulations. 

Asthma 

Four areas of significant improvement in the Asthma simulation suggested echoing was a positive step in 
nursing instruction: Safety and Infection control. Reduction of Risk Potential, Basic Care and Comfort, 

Physiological Adaption, and Psychosocial Integrity. A dependent t-test was performed for each grouping. The 
“ Safety and Infection Control’ first attempt (M = 4.36, std.819) was significantly lower (t (37) = 2.901, p=.006, d = 
0.68) than the echoed attempt (M=4.84, std.546). The “ Reduction of Risk Potential’ cluster demonstrated a second 
echoing (M=15.50, std. = .979) was a significant improvement (t (37) = 4.414, p = .000, <7= 0.80) over the first 
attempt (M= 14.50, std. = 1.465). In the area of ’"Basic Care and Comfort” and significant improvement (t (37) = 
6.033, p = .000, i7=1.16) was demonstrated with the first attempt (M= 0.86, std. = 0.777) being lower than the 
second (M = 1.68, std. 0.619). The fourth area of “Maintance” showed a significant improvement (t (37) =5.808, p 
= .000, d = 0.99). The remaining areas of “Physiological Adaptation”, “Psychosocial Integrity”, and 
“Pharmacological and Parenteral Therapies” all showed increases on the second trial. 

Hypot_Hypoglycemic 

The scenario for “ HypotHypoglycemic ” had four basic areas of investigation with two areas demonstrating 
statistical significance. “ Safety ; and Infection Control” was found to be a significant improvement (t (44) = 3.473, p 
= .001, d = .07) with the first attempt (M = 4.33, std. = .929) found to be less than the second attempt (M = 4.82, 
std. 3 87). Also found was the second trial looking at “ Safety and Infection Control” (M = 5.29, std. 1.07) being 
significantly higher (t (44) = 5.732, p = .000, d = 1.05) than the first (M = 3.67, std. = 1.883). Both remaining areas 
(“ Maintance ”, “ Pharmacological and Parenteral Therapies ”) demonstrated an increase although not statistically 
significant 


Discussion and conclusion 

Imbedding repeating variables in different scenarios provides the students with the opportunity to think through 
varying conditions the proper responses. Some of those variables challenged the participants and their performance 
was lacking, perhaps while concentrating on other tasks. This may be due to increasing complexity of the tasks a 
nurse must perform. It is also possible students didn’t work efficiently to complete more tasks into the allotted 
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fifteen minute time frame. On the whole, HFS repeated attempts showed improvement as they cycled through the 
eight scenarios which supports the findings of others (Dunbar-Reid, K., Sinclair, P., & Hudon, D. 201 1; Fero, L., 
et.al., 2010; Lewis, D., and Ciak, A. 2011). 
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Appendix A 


Creighton Competency Evaluation Instrument (CCE!) 


Scenario: Respiratory Distress 
Pt Initials: C. J. 

PtDx: CHF 

0= Does not demonstrate competency 
1= Demonstrates competency 
NA= Not applicable 

(Circle Appropriate Score for all Applicable Criteria) 

Date: _02_/_19_/_2013_ 
MM / DD / YYYY 

ASSESSMENT 

STUDENT PARTICIPANTS 

Obtains Pertinent Data "are you having difficulty breating" 

0 1 NA 

in two primary nursing roles 

Performs Follow-Up Assessments as Needed "Is your breathing better” 

0 1 NA 


Assesses the Environment in an Orderly Mann^Call light in reach, SR x 2, table in reach 

0 1 NA 

ID: 

COMMUNICATION 

NA 

ID: 

Read Back Order) 

Communicates Effectively with Patient and Significant Other "Explains Oxygen Device" 

0 1 NA 


Documents Clearly. Concisely. & Accurately 



Responds to Abnormal Findings Appropriately - Low Pulse ox 

0 1 NA 


Promotes Professionalism- Identifies self and role as student 

0 1 NA 

FACULTY EVALUATOR 

ID: F- 

CLINICAL JUDGMENT 


Interprets Vital Signs Respiratory rate 

0 1 NA 


Interprets Subjective/Objective Data - Administers NC after assessing 02 Sat 

0 1 NA 


Prioritizes Appropriately - Focuses on Respiratory Distress 

0 1 NA 

Select one of the following 

Performs Evidence Based Interventions - Elevates HOB 

0 1 NA 

□ Clinical 

Provides Evidence Based Rationale for Interventions - assess in debriefing 

0 1 NA 


Evaluates Evidence Based Interventions and Outcomes - assess in debriefing 

0 1 NA 

□ Simulation- initial 

Reflects on Clinical Experience - "1 learned" assessed in debrief 

0 1 NA 

scenario 

Delegates Appropriately 


□ Simulation- repeated 
scenario 

PATIENT SAFETY 

Uses Patient Identifiers - Patient name and DOB 

0 1 NA 

Utilizes Standardized Practices and Precautions Hand Washing 

0 1 NA 


Administers Medications Safely - Begins 02 @ 2L 

0 1 NA 





Performs Procedures Correctly - NC over ears and under chin 

0 1 NA 





COMMENTS 

Permission to use this template must be obtained by the authors. 

This CCEI has been specifically designed for presentation purposes only 
Reproduction of this tool is prohibited by Copyright Laws 

Barbara Gawron has obtained permission by the authors of CCEI to distribute at the Symposium Earned 

If not applicable, circle NA. 

If not applicable, no score is 
given. 

Score = 



For use in The National Simulation Study Revised 7/6/1 1 
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Abstract 

This study investigates student motivation for learning multiplication using different versions of a game- 
based virtual manipulative called Puzzle Blocks. One hundred seventy-eight (N=178) primary students used the 
virtual manipulative software to learn multiplication over ten sessions. Participants’ engagement self-reports were 
collected at three points in time in addition to performance and learning measures. The study provides evidence that 
design decisions impacting student interactions within the virtual manipulative environment have a direct impact on 
student motivation and learning. 


Introduction 

Physical manipulatives are used in math education to introduce abstract concepts to young children. They 
are often deployed based on the assumption that activities involving physical objects are more engaging and 
effective for young children because they make learning more concrete and therefore understandable. With the 
advent of technology, physical manipulatives have been transformed into virtual manipulatives: interactive visual 
representations of dynamic objects that present opportunities for constructing mathematical knowledge (Moyer, 
Bolyard, & Spikell, 2002). Virtual manipulatives are promising educational tools with many advantageous 
properties (see Durmus & Karakirik, 2006; Suh & Moyer, 2007; Suh, Moyer, & Heo, 2005). Importantly, studies 
have shown that instruction with virtual manipulatives can increase student motivation for learning math (Bouck & 
Flanagan, 2010; Durmus & Karakirik, 2006; Steen, Brooks, & Lyon, 2006). 

The idea that virtual manipulatives can influence student motivation in mathematics raises a number of 
important questions. For example, does the motivation elicited through virtual manipulative use lead to higher 
learning outcomes? Do students who are more successful with virtual manipulatives have higher motivation for 
learning math compared to students who are less successful? In other words, how do virtual manipulatives influence 
the motivation of different types of learners? These important questions are worthy of examination as students’ 
experiences with virtual manipulatives have been shown to impact learning outcomes (Paek, Floffman, & Black, 
2013a; Paek, Floffman, Saravanos, Kim, & Black, 2012) and may help shape their expectancies and values for the 
domain of math (Eccles & Wigfield, 1995). 

Thus, this study examines students’ motivation for learning mathematics using a game-based virtual 
manipulative. It argues that learning activities situated in a rich multimedia context can be designed to foster student 
engagement that may ultimately improve learning outcomes. 

Theoretical Background 


Mathematics Achievement and Motivation 

It is well-known that American children tend to enjoy mathematics in the primary grades but that their 
enjoyment declines dramatically as they progress into high school (Middlton & Spanias, 1999). Considering the fact 
that students’ achievement in mathematics also declines as students go from the primary to the secondary grades, 
there have been many studies examining the relationship between students’ math achievement and their motivation 
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for learning math. For example, Gottfried, Marconlides, Gottfried, Oliver, and Guerin (2007) argue that math 
achievement is a significant contributor to the decline in intrinsic mathematics motivation from childhood through 
adolescence. Other studies have shown the importance of motivation for predicting students’ academic achievement. 
Steinmayr and Spinath (2009) examined how different motivation constructs such as achievement motives, goals, 
ability self-perception, values and school performance, incrementally contributed to the prediction of school 
achievement for 1 1 th and 12 th grade students, and found domain-specific ability perceptions and values showed the 
highest increments. Shores and Shannon (2007) investigated the relationship between self-regulated learning, 
motivation, anxiety, attributions and achievement in mathematics. The results of their work show that motivation 
and anxiety have a significant impact on fifth grade students’ test scores and mathematics grades. 

There are also studies examining younger students’ motivation for learning and math achievement. Arnold, 
Fisher, Doctoroff, and Dobbs (2002) found that preschool-aged children, who participated in a 6-week classroom 
intervention integrating math-relevant activities into their daily routine, scored significantly higher on a standardized 
test of math. They also enjoyed math activities more than children who did not received the intervention. DiPerna, 
Volpe, and Elliott (2005) tested a model of the relation between motivation, interpersonal skills, engagement, study 
skills, and mathematics achievement using structural equation modeling. They found that motivation predicts study 
skill and engagement, which are positively associated with academic achievement. 

Theses studies show the strong connections between students’ motivation for learning mathematics and 
their achievement in the domain, even though the causal relationships are not fully understood. Together these 
studies suggest that providing learning activities that can motivate students to engage with mathematics might be a 
means to boost student achievement. 

Game-Based Learning Environments 

Game -based learning continues to be a topic of interest in educational settings. Although these discussions 
often begin with concerns over the possible negative influences of video games, such as aggressive behavior, 
violence, and physiological arousal (Anderson & Bushman, 2001), more recent perspectives tend to view games as 
potentially valuable vehicles for learning. Such work often considers the beneficial aspects of games and gaming in 
their various contexts, and how these elements may promote learning. For example, student engagement, 
collaboration, goal-based learning, scaffolding, and interactivity have all been topics in the game-based learning 
literature (Garris, Ahlers, & Driskell, 2002; Gee, 2003; Mayo, 2009; Squire, 2003). In particular, affective constructs 
such as engagement, enjoyment, and motivation in game -based learning environments have been regular research 
topics. For example, emphasizing engagement with games, Prensky (2001) explains that specific aspects of games 
such as fun. play, rides, and goals, make them engaging. He argues that it is important to have fun during the 
learning process, since fun creates relaxation, which enables a learner to “take things in” more easily. He also 
suggests that the motivation players feel while playing games enables them to put forth effort without resentment. 

Other work has connected both game play and mathematics. For example, a study conducted by Rosas et 
al. (2003) shows that first and second grade students, who played computer games for 30 hours during three months, 
showed increased motivation for learning mathematics. Another study by Ke and Grabowski (2007) investigated the 
effects of game play on fifth grade students’ mathematics performance and attitude. The findings suggest that game 
play was more effective than drill and practice in promoting mathematics perfonnance, and that cooperative game 
play had the most effect on promoting positive math attitudes. Together, such studies recognize the value of student 
engagement, positive attitude, and motivation towards game-related activities. In addition, they suggest that math- 
related games might be a way to promote student motivation for the domain of math. 

Game-Like Virtual Manipulatives 

Since student engagement is often cited as a reason for the use of manipulatives in mathematics education, 
and given the well-documented motivational aspects of games, it may be possible to create a game-like virtual 
manipulative environment that combines the desirable features of games and with the educational underpinnings of 
manipulatives. With this idea in mind, the researchers developed a game -based virtual manipulative environment, 
called Puzzle Blocks. Puzzle Blocks was designed to introduce the concept of multiplication to young children 
through repetitive physical activities in a rich, multimedia environment. While using the virtual manipulative, 
students experience the concept of multiplication visually, aurally, and physically. 

Since Puzzle Blocks was designed as a game-like virtual manipulative, it was important to examine how 
playing with the software would impact young children’s understanding of multiplication and how various 
multimodal interactions mediated any learning outcomes. After working with first and second grade students, 
researchers found that playing Puzzle Blocks for ten sessions over a six week period led to students’ learning 
outcomes as measured by paper-based tests. The results also revealed that providing auditory feedback and allowing 
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the manipulatives to be controlled though a touch-based interface contributed to student learning (Paek et al., 2013a; 
Paek, Hoffman, & Black, 2013b). The finding that playing Puzzle Blocks led to gains in student learning, and that 
different sensory experiences seemed to influence their understanding, it seemed reasonable to examine the impact 
of the various versions of the virtual manipulative environment on student motivation. Thus, key questions in the 
current work include: Do students enjoy playing Puzzle Blocks ? Does their enjoyment impact learning? Do different 
learning outcomes lead to different self-reports of engagement? This paper presents the results of students’ 
motivations throughout their participation in the previous study. 

Method 


Research Instrument 

Puzzle Blocks 

Puzzle Blocks is designed to introduce the concept of multiplication in a game-like manner. The game 
mechanic for playing Puzzle Blocks is building objects such as a rocket ship or a train with smaller blocks. For 
example, in one level dealing with the two times table, students have to build a house. This house is composed of 
groups of blocks of various lengths that are all multiples of two (four, six, ten, twelve, so on). Each of these larger 
groups is referred to as a “target group.” To build the house, students need to construct the target groups by 
repeatedly combining smaller groups of two blocks, referred to as “source blocks.” Once the software highlights a 
target group, students are charged with filling that group with source blocks. When a target group is filled with the 
correct number of source blocks a hidden piece of the background is revealed. The more target groups students 
complete, the more of the background they can see. Finally, when all of the target groups are built, the entire 
“puzzle” becomes visible and the level is complete. 



Figure la. The two times table, level Figure lb. A target group and a Figure 1c. The two times table, level 

one: starting with an object (a house) source block. Students drag source one, at completion: the completed 

that needs to be built. blocks to the highlighted target age-appropriate puzzle. 

group. 


Visual and Auditory Feedback 

Importantly, while students move the on-screen blocks, the game provides visual feedback. The visual 
feedback consists of both graphic feedback as well as symbolic feedback. For example, to build a target group of six 
blocks in the two times table, students have to move source blocks with two units three times. While the student 
moves the source blocks to the target group, he or she can receive two types of visual feedback. The first type is 
graphical as it shows the initially empty target group being filled with source blocks (two at a time). The second 
type of feedback is symbolic as it shows the result of student actions using standard mathematical notation such as 
“x 1,” “x 2,” or “2x3= 6.” The symbolic feedback is aligned, spatially and temporally, to the corresponding 
graphical feedback in an effort to help learners connect their actions with the manipulatives to the resulting 
mathematical notation. 
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Figure 2. Visual feedback: The graphical feedback (the blocks) and the corresponding mathematical 
notation (i.e., “x 1” or “2x3 = 6”). 


Along with its visual feedback, Puzzle Blocks provides auditory feedback through a narrative voiceover that 
“reads aloud” the information contained in the symbolic feedback. This voice over is provided contiguously with the 
visual math notation. For example, when a target group has a length of nine units and the source block has a length 
of three units, a student has to move the source blocks three times (3x3 = 9). Each time a player grabs a new source 
block, the number “3” is presented visually. Simultaneously, the software’s narrator says, “three.” As the source 
blocks are moved to the target group by the player the mathematical notation counts up based on the number of 
times a group of source blocks has been added, resulting in visual feedback that reads “x 1,” “x 2,” or “x 3”, and so 
on. Each time this feedback is updated, the narrator says “times one,” “times two,” and “times three.” Finally, when 
a target group is completely filled with source blocks, the resulting mathematical equation appears (i.e., “3x3 = 9”) 
and the narrator says, “three times three equals nine.” 

Levels for Scaffolding 

To introduce the concept of multiplication to young children, the software draws on learners’ previous 
knowledge of addition and gradually scaffolds learners toward the target concept using five distinct levels. As 
mentioned previously, the main mechanic of the environment is centered on completing puzzles by building larger 
groups of blocks out or smaller groups. This mechanic remains consistent, however, there are slight variations in 
each of the five levels. Importantly, every times table uses the same five-level structure. 

In the earliest levels (1 - 3), students build blocks by simply moving source blocks to the target group. This 
somewhat repetitive experience is reinforced with visual and/or auditory feedback. The only difference between 
these first three levels is their emphasis. In level one, the emphasis is on repeated addition, and students are shown 
corresponding mathematical notation (i.e., 2 + 2 + 2 = 6). In level two, students stack the source blocks horizontally, 
but the emphasis is on the number of times the source blocks have been added. Again, students are exposed to the 
corresponding notation, “x 1” or “x 2,” and eventually the entire equation, “3x4= 12.” Level three is similar to the 
second level but students stack the blocks vertically which links to how one might calculate the area of square. 

By level four, students can no longer build the target group by moving source blocks; instead they must 
now decompose the target group in order to determine how many source blocks it can hold. For example, to fill a 
target group with ten units, instead of moving a source block of two units five times, students have to think about 
how many times they have to move a unit of two. Students are shown the corresponding problem in mathematical 
notation (i.e., “2 x ? = 10”). As shown in figure 3a, students use a numeric stepper to select their answer. They then 
drag that answer onto the corresponding equation. If the number is correct, the target group is filled with the 
appropriate color and a hidden image is revealed. In the final level (Level 5), students also need to complete 
equations to build target groups, but in this level, students need to find the product of each problem presented (i.e., 
“2x5 = ?”). 



Figure 3a. The two times table, level 
four: Using a numeric stepper, students 
can complete the equation “2 x ? = 4” 



Figure 3b. The three times table, level 
five: Using a numeric stepper, students 
can complete the equation “3x3 = ?” 
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Research Design and Participants 

For research purposes multiple versions of the virtual manipulative environment were deployed. This 
allowed the researchers to systematically vary various design features of the software while holding other aspects of 
the experience such as the number of problems, the sequence of the problems, and the problems themselves 
constant. In total, six different versions of the software were tested. Four of these versions were based on variations 
of two factors: the presence of audio narration (Present vs. Not-Present) and input method (Touchscreen vs. 
Computer Mouse). All four of these groups saw relevant mathematical notation while interacting with the on-screen 
manipulatives, however, what they heard was either game-like sound effects or audio narration. The groups were 
further delineated by the input device used to control the on-screen blocks. Together, these four versions of the 
software were united by the fact that they all presented mathematical notation. Thus, these four conditions were 
considered the “math” groups. 

In contrast, two more groups also experienced the on-screen blocks but did not see any math notation. 
These two conditions were considered the “game” groups. These groups heard sound effects but no audio narration 
and either moved the blocks with a computer mouse or with their fingers on a touchscreen. All other aspects of the 
software were the same as the other versions. Essentially, participants in these groups experienced the same 
“puzzles” and enacted the same movements as the math groups but without any reference to mathematics. Figure 4a 
and 4b illustrate how Puzzle Blocks looked with (4a) and without (4b) mathematical notation. 



Figure 4a. Examples from the math 
group, which included standard 
mathematical notation. 



Figure 4b. Examples from the game 
group, which made no reference to 
mathematics. 


Participants 

Eighty-eight first grade students and ninety, second grade students (N = 178), from NYC schools, were 
assigned randomly to one of six groups. Among the math groups, participants in the M-AT group (N=38) received 
audio narration and used a tablet computer. The M-T group (N=37) did not receive audio narration but used a tablet 
computer. The M-AM group (N=33) received audio narration and used a computer mouse. The M-M group (N=32) 
did not receive audio narration and used a computer mouse. Lastly, the two game groups, G-T (N=19) and G-M 
(N=20), played Puzzle Blocks without seeing any mathematics content or hearing any audio narration. The G-T 
group moved the blocks with their finger on a touchscreen and the G-M group used a computer mouse. All six 
groups used the virtual manipulative environment for ten sessions over six weeks. Each session lasted an average of 
twenty minutes. 
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Tablet 

Laptop 


Table 1. Number of Participants by Group 

Math Groups Game Groups 

Audio Narration No-Audio Narration No-Audio Narration 


M-AT (N=38) M-T (N=37) G-T (N=19) 

M-AM (N=37) M-M (N =32) G-M (N=20) 


Materials 

To measure student motivation, a brief, age -appropriate survey was administered at three different times: 
once prior to playing Puzzle Blocks for the first time (pre), once after completing the fifth play session (mid), and 
once after the tenth and final play session (post). The pre- and mid-surveys were based on the Intrinsic Motivation 
Inventory, and included questions in four categories: interest/enjoyment, perceived competence, effort/importance, 
and value/usefulness. The questions were modified slightly in order to make them more age appropriate. In addition, 
the number of questions was reduced. The pre-intervention survey included 12 questions asking participants about 
their motivation for learning math. Questions included: How much do you enjoy math? How good are you in math? 
And, how helpful do you think math is? The mid-intervention survey was similar to the pre-survey but this time the 
questions focused on Puzzle Blocks. Questions asked included: How much do you enjoy playing Puzzle Blocks ? 
How good are you at playing Puzzle Blocks ? And, how helpful do you think playing Puzzle Blocks is for learning 
math? Participants’ average score for the 12 questions was used as a measure of their motivation for learning math 
(pre-survey) and playing Puzzle Blocks (mid-survey). 

The post-intervention survey had seven questions and asked participants about their perception of learning 
through Puzzle Blocks. Questions included: How much did you learn about multiplication from playing Puzzle 
Blocks ? How good do you think you are at playing Puzzle Blocks ? And, how much do you want to continue playing 
Puzzle Blocks ? Again, the participants’ average of the seven questions was used as a measure of their motivation. 

For each motivation survey, questions were presented as statements and participants were asked to respond 
by indicating how much they agreed or disagreed with each statement. Participants were provided a four-point 
Likert-scale, consisting of child-friendly “smiley” faces representing “I strongly disagree,” “I disagree,” “I agree,” 
and “I strongly agree.” Given the age of the participants, no text was shown on the scale. Participants were asked to 
mark the face that matched their level of agreement. 



Figure 5. Foun-poiint Likert-scale used on the motivation surveys. 


Results 

The pre-intervention survey asked participants to self-report their motivation for the domain of math. A 
total of 178 participants (100%) responded to the survey. The mean score of their responses, on a four-point scale, 
was 3.19 ( SD = .48). To examine the differences between the six groups, an Analysis of Variance (ANOVA) was 
conducted. The results found no significant difference between the six groups, F (5, 172) = .225, p = .951. As no 
differences were detected at this level, no further analysis was warranted. 

After five sessions of playing Puzzle Blocks, participants took a mid-intervention survey asking about their 
motivation for playing Puzzle Blocks. Overall, participants reported a high level of motivation with an average of 
3.50 our of 4 (SD =.49). As shown in Table 2, the M-AT group had the highest mean score (M= 3.64, SD = .40), 
and the G-T group had the lowest mean score (M— 3.31, SD = .51). The result of an ANOVA revealed that the 
difference between the groups’ self-reported motivation scores for playing Puzzle Blocks was marginally significant, 
F (5, 172) = 2.023, p = .078. 
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Table 2. Mean and Standard Deviation of the Mid-Intervention Motivation Survey by Group 



Group 

N 

M 

SD 

Math 

M-AT 

40 

3.64 

0.40 


M-AM 

34 

3.49 

0.65 


M-T 

37 

3.59 

0.33 


M-M 

30 

3.39 

0.54 

Game 

G-T 

17 

3.31 

0.51 


G-M 

20 

3.38 

0.38 


After playing Puzzle Blocks for ten sessions, 176 participants completed the post-intervention motivation 
survey. Again, the overall means reported were quite high (M= 3.52, SO =.53). However, significant differences in 
participants’ self-reported motivation were found, F( 5, 170) = 3.977. p = .002. A Tukey’s HSD post-hoc test 
revealed the mean motivation score for the math groups, M-AT (M= 3.56, SD = .55), M-AM (M= 3.63, SD = .33), 
M-T (M = 3.63, SD = .38), and M-M (M= 3.56, SD = .64), were significantly higher than the motivation score for 
the game group, GM (M =3.10, SD = .58). 


Table 3. Mean and Standard Deviation of Post-Intervention Motivation Survey by Group 



Group 

N 

M 

SD 

Math 

M-AT 

40 

3.56 

0.68 


M-AM 

34 

3.59 

0.46 


M-T 

36 

3.64 

0.39 


M-M 

30 

3.55 

0.69 

Game 

G-T 

17 

3.34 

0.61 


G-M 

19 

3.17 

0.56 


Participants’ responses to the three motivation surveys were also analyzed in conjunction with their 
learning outcomes. As described earlier, learning outcomes were measured using a paper-based test designed to 
probe students’ conceptual understanding of multiplication (Paek, 2012). Recall that significant differences were 
found — both at mid-test and post-test. Thus, the final step of the analysis involved an examination of the correlation 
between participants’ motivation self-reports and their learning outcomes. No correlation between participants’ 
motivation for playing Puzzle Blocks and their mid-test score was found. Also, no correlation was found between 
participants’ post-intervention motivation scores and their post-test scores. 

Conclusion 

Because of their emphasis on physicality, manipulatives are a popular tool used in mathematics education. 
While many studies have discussed on how manipulatives can assist young children in understanding abstract 
concepts, other work has emphasized the motivational aspects of allowing students to work with concrete 
manipulative objects. Since there is a positive correlation between student motivation for learning math and their 
mathematics achievement, the current work argues that providing engaging learning activities is a critical aspect of 
mathematics learning. Thus, this paper examined student motivation for learning math using a custom virtual 
manipulative environment. While participants used this environment for ten sessions, motivation self-reports were 
taken at three different points in time (pre, mid, and post-intervention). 

All participants took a pre-intervention survey about their motivation for the subject of math before playing 
Puzzle Blocks. Overall, students reported high motivation for math, and no significant differences were found 
between groups. After five sessions of play, students took a mid-intervention survey asking about their motivation 
for playing Puzzle Blocks. The results showed high motivation for playing Puzzle Blocks across groups. Marginal 
differences between groups were found, F (5, 172) = 2.023, p = .078, however, the range of mean scores between 
the highest (M-AT: M— 3.64, SD = .40) and the lowest (G-T: M— 3.31 SD = .51) was .33. Lastly, participants 
completed a post-intervention survey asking about their experience with Puzzle Blocks. The results showed 
significant differences between groups, F (5, 170) = 3.977, p = .002. Further analysis found that the versions of 
Puzzle Blocks that included mathematical content elicited higher levels of motivation from students compared to 
students who used Puzzle Blocks as a pure game devoid of instructional content. Furthermore, the differences 
between the four math groups (M-AT, M-AM, M-T, and M-M) and the game group (G-M) were statistically 
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significant. Although the difference between the four math groups and the other game group (G-T) were not 
significant, the G-T group’s mean (M = 3.38, SD = .38) was lower than all four of math groups. 

Taken together, these findings support the notion that well-designed learning activities can be as 
motivating as, or, in this case, more motivating than, non-educational game experiences. Although further research 
is needed to confirm these findings, it is an encouraging outcome for educators, instructional designers and 
researchers. It also highlights the importance of carefully designed digital learning environments where multiple 
factors may be influencing the learners’ experience. 

The researchers recognize the limitations of this study. First, the work did not find any correlation between 
participants’ motivation for learning multiplication by playing the game-based virtual manipulative and their 
learning outcomes. This result might be due to the fact that the measures of motivation were all based on participant 
self-reports. Given the young age of the participants it is possible that they may have exaggerated their actual 
engagement with the activity in an effort to please the researcher or to continue using the novel software. Indeed, 
many researchers have pointed out that current motivation studies are overly reliant on self-report methodologies, 
and need alternative approaches (Fulmer & Frijters, 2009). Therefore, in terms of future work, additional approaches 
to measuring participant motivation should be considered. 

Still another limitation includes the fact that the number of participants in each group was different. An 
ideal study would have the same number of participants in each experimental group. Given the smaller sample size 
of the two game groups, the results should be interpreted conservatively. Lastly, there should be further investigation 
into changes in student motivation over a longer period of time. Although student participation in a six-week study 
is not trivial, future work should consider investigating the impact of virtual manipulative use across an entire 
academic year. 
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Abstract 

Research looking at students completing virtual science labs is substantial in physical brick-and-mortar K- 
12 classrooms and the college setting. However, it is lacking when students are completing these labs in a blended or 
cyber charter K-12 school. This interpretive, phenomenographic study explored the perceptions that students hold 
regarding their virtual labs in a blended charter school. Four students were interviewed to discuss their 
understanding and perceptions that they have of completing their science labs virtually in a blended school. Working 
from a theoretical framework of constructivism, community of inquiry, and 21st century literacy the data was 
analyzed. This framework helps to situate the work in the context of the learning environment and include the skills 
that students in these schools are using on a daily basis. The findings from this pilot study tend to refute some of the 
commonly held views of virtual labs regarding their interactivity and authenticity to physical experiments. 

However, the value and alternative worth of virtual labs is a conception held by two participants. In addition to this, 
the communication and collaboration strategies of participants varied with which information processing and 
clarification methods they used to understand the material. 

Introduction 

Cyber charter schools and blended charter schools are a relatively recent development in the public school 
landscape that offer a choice to traditional public schools with consistent increases in enrollments since they first 
appeared. In the 2011-2012 school year there were approximately 275,000 students enrolled in a fully online school 
in 31 states (Watson, Murin, Vashaw, Gumin, & Rapp, 2012). In Pennsylvania, 32,322 students attend 16 cyber 
charter schools, some of which enroll regionally and others statewide. (Watson et ah, 2012). While these schools are 
expanding within several states, there is not a significant amount of research on the effectiveness or success of them. 
Even the people who have researched the effectiveness have not focused on specific subjects (Darrow, 2010; 
Hubbard & Mitchell, 2011; Zimmer et ah, 2009). In addition to totally virtual schools, there are a certain number of 
charter schools that follow a blended model. In blended schools, students receive some of their instruction in a 
physical location, but also use the computer for delivery of at least some of their instruction. The school selected for 
this study, Tullahoma Blended Charter School, follows an interesting model in which some students stayed home 
and others went to the school but still interacted with their teacher through the computer. These students rely heavily 
on the computer for the majority of their coursework and yet the current research does not focus on their experiences 
with computer simulations, and rather researches students completing these virtual labs while still in the context of a 
direct and physical relationship with their teachers and peers. 
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Learning does not occur in a vacuum and “situated cognition that emphasize the reciprocal character of the 
interaction in which individuals, as well as cognition and meaning, are considered socially and culturally 
constructed” (Lave, 1988, 1993; Collins, 2006; as cited in Barab & Duffy, 2012, p. 26) emphasize the construction 
of knowledge that is contingent on the context of the learning environment. Therefore, the studies that have 
compared virtual and physical laboratories that have been conducted when students are present in the classroom 
(Pyatt & Sims, 2011; Klahr, Triona, & Williams, 2007) are not relevant to direct transfer in online learning 
environments. Tullahoma students are situated in a variety of contexts and learning environments and come from a 
diverse geographic region. For example, students in this school are likely to come from different school districts and 
thus have a wider range of prior knowledge than students who are relatively static in the same district. Blended 
schools that use the computer as their main delivery system typically utilize simulations to help students learn and 
grasp concepts through modeling or visual representations. 

The Next Generation Science Standards (NGSS) Nature of Science appendix states that one of the 
fundamental goals “for K-12 science education is a scientifically literate person who can understand the nature of 
scientific knowledge. (NGSS Appendix H, 2013, p. 2). The underlying ideas that students should take away from 
science to apply to their lives as an educated citizen that relate closely to virtual simulation are “scientific 
investigations use a variety of methods, scientific models, laws, mechanisms, and theories explain natural 
phenomena, scientific knowledge is open to revision in light of new evidence, and science addresses questions about 
the natural and material world” (NGSS Appendix H, 2013, p. 4). Effective science instruction that helps students 
take away an integrated understanding of these skills increases the probability that students will apply the 
knowledge gained in their science education later in life. In the College and Career Readiness appendix, this 
knowledge of science is tied into being successful later in life by stating that “an education grounded in acquiring 
and applying knowledge positions students to improve their options in a rapidly changing menu of jobs” (NGSS 
Appendix C, 2013, pgs. 1-2). According to a study by Millennial Branding Inc., the most important skills to 
employers that are hardest to find in students are communication skills (91%), the ability to adapt to change (85%), 
and strategic thinking and analytical skills (78%) (Schwabel, 2013). 

Studies comparing virtual and physical laboratories have been conducted when students are still physically 
present in the classroom (Pyatt & Sims, 2011; Klahr, Triona, & Williams, 2007) have found that students can 
experience conceptual change in both ways either by ‘hands-on’ activities or using models. There are also many 
studies that demonstrate student attitudes towards learning and conceptual change can impact their learning gains 
while completing their labs (Winn et al., 2006). An inquiry-based lab fielded in the pedagogical theory of 
constructivism can evoke more curiosity and learning gains than traditional teacher-led classrooms (Tsai, 1998). One 
of the goals of the laboratory experience is for students to develop teamwork abilities (Singer, Hilton, & 
Schweingmber, 2006). Due to the novel structure of Tullahoma students may have a different experience based on 
their location and ability to access peers and the teacher. This study fills a gap in the literature by examining in an 
in-depth way the collective perceptions of students who are completing labs in this innovative manner and the 
variations in student experience based on past experiences. 

Theoretical Framework 

Constructivist based learning environments evoke concepts such as authentic learning, problem-based 
coursework, ownership of inquiry, scaffolding, the zone of proximal development, metacognitiveness, knowledge 
construction, and reflection on learning (Duffy & Jonassen, 1992; Fosnot & Perry, 1996; Jonassen, 1999; Jonassen 
2013; Mayer, 1999). The above characteristics should be visible in any well-designed science course, whether face- 
to-face or online. In addition to these, effective science instruction is seen as allowing students to work through 
scientific concepts and build understandings similar to how scientists in the field operate (Bransford et al., 2000; 
NGSS - Appendix C, 2013). Constructivism states that “there are many ways to structure the world and that there 
are many meanings or perspectives for any event or concept” (Duffy & Jonassen, 1992, p. 3). An individual’s 
“present conceptions are products of diverse personal experiences, observations of objects and events, culture, 
language, and teachers’ explanations” (Anderson, Lucas, & Ginns, 2003). Situated cognition emphasizes this idea by 
incorporating the context of learning into the knowledge structure and experience of individuals in that environment 
(Barab & Duffy, 2012; Durning & Artino, 2011). The activity and setting in which people learn cannot be removed 
from the cognitive process of learning and the situations can be said to be a co-influence the learning process and 
experience of a student (Brown, Collins, & Duguid, 1989). Students who are completing their science course work 
in a blended school have flexibility in how and when they complete their labs and likely will ascribe several 
different meanings of the labs that they experience based on prior experiences, scientific conceptions, scaffolding, 
and feedback. 
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Lave recognized a community of practice as a group that helps to build a meaningful learning system that 
implies participation in an activity system “about which participants share understandings concerning what they are 
doing and what that means in their lives and for their communities (as cited in Barab & Duffy, 2012, p. 40). The 
tools that a culture uses help to define the learning that takes place by contextualizing the tools (Brown, Collins, & 
Duguid, 1989). The online science students at Tullahoma use the computer and simulations as mindtools to help add 
meaning to their learning. A community of inquiry is a popular notion within higher education online environments 
that helps to support collaborative learning and helping learners to perceive what they are learning (Garrison & 
Arbaugh, 2007) and “it provided a collaborative-constructivist perspective to understanding the dynamics of an 
online learning experience” (Arbaugh et al., 2008). The community of inquiry is also appropriate for students in K- 
12 online learning environments because of the systemic and nonlinear view of learning and the popular model of 
conceptualizing the online learning experience. The following figure shows the parts of the learning environment 
that affect an online community of inquiry. It has been adapted for the context of the students at Tullahoma charter 
school. 



Figure 1 : A preliminary adaption of the community of inquiry framework oriented for K-12 students in 
online schooling. Modified from Garrison and Arbaugh’s (2007) work on the COI 

Blended charter school students use the computer frequently to access their coursework, communicate with 
their teachers and peers, complete their assignments, and view videos and interactive tools. The computer is 
essential to their schoolwork and their ability to successfully navigate the curriculum. These students are gaining 
valuable computer skills and “Information literacy is conceivably the foundation for learning in our contemporary 
environment of continuous technological change” (Bruce, 2004, p. 8). Since Tullahoma blended charter school 
students receive all of their instruction via the computer they are engaging with information and communication 
technologies (ICT) on a regular basis. This important skill to have in the 21 st century is something that virtual labs 
can help to develop by having students interact with different software, different communication platforms, and 
navigating between multiple pages and integrating many sources of information. Additional 21st century skills such 
as self-directed learning, collaboration, problem-solving, and information seeking and knowledge building through 
strategies like Just-in time teaching. Online classes utilize many web 2.0 tools and the science class selected in this 
study has synchronous classes, video technologies, lessons sequentially arranged online, and interactive 
demonstrations. As mentioned previously, building teamwork and collaboration is an important part of science labs 
(NSTA, 2007, Singer et. al., 2006). The research questions posed for this pilot study are: 

• How do students perceive their learning in these laboratory investigations related to the structure of the 
activities? 

• How do students make sense and understand the activity that they are completing? 
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Research Methods 


To build structure for understanding student experience with their online virtual simulations in a blended 
charter school this study employed an interpretive, phenomenographic methodology. Phenomenography aims to 
examine the variations in the world as people experience it and to adequately explain this to other’s who have 
different experiences with the world (Sjostrom & Dahlgren, 2002). According to Marton, phenomenography aims to 
“deal with both the conceptual and the experiential, as well with what is thought of as that which is lived. It also 
deals with what is culturally learned and with what are individually developed ways of relating ourselves to the 
world around us.” (1981, p. 181). This research approach is appropriate for this study because students will be 
directly asked about their learning experiences with their science labs and knowledge will be viewed from a second- 
order perspective. The units of analysis will be the various collective conceptions that students form about their 
virtual labs (Marton & Pong, 2005). The phenomenographic approach can then be used to give a collective 
foundation for the perceptions’ that students ascribe to their lab work. 

Description of Research Settings 

For clarity, a definition of simulated labs is provided: “simulated labs are the imitations of real 
experiments. All of the infrastructure required for laboratories is not real, but simulated on computers” (Ma & 
Nickerson, 2006). By real, they refer to the need to have physical materials and equipment in the lab to carry out the 
experiments that the simulations are demonstrating. Virtual labs do not necessarily have all of the components of a 
physical lab. They can be altered to eliminate procedures, materials, tasks, or features that would not aid in helping 
the students to undergo conceptual change. Wieman and Perkins (2005) realize that “a real-life demonstration or lab 
includes enormous amounts of peripheral information that the expert instructor filters out without even thinking 
about it” (p. 36).Using virtual labs with specific features enhanced and others hidden can attempt to alter the 
perception that students leam from the labs so that they walk away with the intended conceptual change as defined 
by the developers of the lab. The software chosen for this school is Odysseyware which “provides innovative, 
Elearning solutions to schools nationwide. With more than thirty years of experience behind us, we offer 21 st 
century learning opportunities for today’s digital natives” (Odysseyware, 2013). 

This research took place within the context of a blended charter school using the computer as an active 
communication medium (Winn et al., 2006). Tullahoma features a hybrid learning model. There are two physical 
buildings that the students can go to and then other students can log in virtually and attend school in a traditional 
‘cyber’ sense. Even the students who are in the same room with their teacher are logged in through a virtual 
classroom and interact via the platform. This is an ideal setting to answer the research question because there is a 
significant amount of literature comparing virtual and physical labs within the traditional brick-and-mortar school, 
but little to none on how learning happens when the students are completing these labs in different locations and 
communicating with their instructor and peers through synchronous and asynchronous virtual interactions. In a 
study by Pyatt & Sims (2011) that examined the learning dimensions that occur through physical and virtual 
laboratory work. This study and others like it have not looked at labs that occur in blended cyber charter schools. 

The NSTA advises that blended schools use “a range of active scientific investigative experiences should be 
integrated into the instructional process for all students” (NSTA, 2007). Tullahoma uses the computer as their main 
delivery method; the students view a virtual lab and then can complete a lab activity via the computer. The lab is one 
way to incorporate different activities encompassing the learning of science as recommended by the NSTA. 

Sampling Strategy 

Criterion, purposive sampling with an emphasis on variation was chosen for this study. According to Creswell 
(2013) criterion sampling allows for rich and deep data to be collected because the participants selected will have 
the experiences necessary to elucidate their understandings. Variation is important because the goal in 
phenomenography is to have multiple outcome spaces in the data analysis phase so that all of the possible collective 
ways of relating between the world and one’s own ideas will be exemplified (Akerlind, 2005). The criteria for 
inclusion are that the participants have been enrolled in Tullahoma for at least half a semester and that they have an 
assignment completion rate of 80% in their science class. Once students meet these criteria, further criteria of 
geographic diversity and past experience in different school environments will be preferred, but not necessary. Four 
students agreed to be interviewed for this study. Two of the students are not in the same site as the teacher, one 
works from home, and one splits their time between the teacher’s site and being home. They are all in biology class 
and regularly participate in virtual labs. 


556 



Data Collection 

In phenomenography, the main source for data collection is open-ended interviews (Larsson & Holmstrom, 
2007). The theme for the interviews was to have some preformed questions to get the students talking about and 
reflecting on their experience with the virtual labs. The questions should be grounded in what the students are saying 
about their experience to truly see what this is like for a particular student. 

I conducted four hours of observation to gather a sense of the structure of the course and how students 
interact and also to introduce myself. I conducted one interview with each of them over the course of two weeks. It 
has been shown that 20 participants are enough to discover all of the different ways of understanding the 
phenomenon (Larsson & Holmstrom, 2007). My four participants are a pilot study for future research with a greater 
sample. The interviews were semi-structured with guiding questions and lasted between one hour and one hour and 
fifteen minutes. I conducted these interviews in a private AdobeConnect virtual classroom. Since all of the 
interviews involved typing, there were grammatical errors from the participants. To maintain accuracy and 
authenticity to the original interviews, I have not corrected spelling or grammatical errors on the part of the students. 
These interviews were recorded and a link was created that allowed me to access the recordings for transcription. 

Data Analysis 

To conduct the analysis I followed the steps recommended by Sandberg (2000). The recommended process 
is to first familiarizing yourself with the interview notes and transcriptions, then engaging in a back and forth 
process identifying what occurs in the phenomenon (virtual labs) being studied and the variation in how it is 
understood by participants (what they focused on in the virtual labs) with the context in mind. This iterative process 
should first be completed for individual participants and then comparing these understandings with cross-case 
comparisons to make clearer the similarities and differences across perspectives. The last step in the analysis is to 
join the what and how into one unit in order to create the hierarchical categories in which full variation and 
understanding of the phenomenon are derived. Since there are only four participants in this study it is realized that 
the full variety of understanding will not likely occur. The categories are likely to be rudimentary and would be 
refined and developed through more analysis and interviews with more participants. 

Researcher Positionality 

Originally, this study was going to be conducted at a larger cybercharter school in Pennsylvania with a 
sample size between 10-20 students to align with the recommended criteria for a phenomenographic study to 
explore the individual perceptions in enough depth to arrive at well-developed collective experiences (Larsson & 
Holmstrom, 2007). After getting initial approval from the administration of the school and being put in contact with 
a middle school teacher, it was discovered that the students did not do virtual labs and thus made an inappropriate 
cohort to study for the purpose of this research. This helped to transition the paper to a pilot study as a baseline for 
future work. 

To locate a new site, I activated my personal network and conducted this research at a former place of 
employment and thus already had a connection with the science teacher. To come in contact with the students I first 
let the teacher, Mr. Smith know what my sampling requirements were and to have him announce the research in 
class to gauge an interest level. Once this was complete I entered the Biology class and introduced myself and the 
goals of the study. I got the four participants interested from Mr. Smith and told the students what type of research 
this was, what would be required of them, and gave them my contact information. I realize that the students who 
know me may answer differently than those who I am meeting for the first time since I already have a rapport 
established. The first half of one of the interviews was conducted as a two person interview since both of these 
students were ahead in their work, work from the same site, and shared a room for the interview. This could lead to 
an ethical issue were there answers may have been influenced by the other student and thus a full variety of 
understanding may not be understood. However, the second half of the interview was conducted separately and the 
main theme and type of responses I was getting from each student were not different from those in the first half of 
the interview. 

Research reflexivity in interpretive work is necessary for addressing the subjectivity and biases that are 
brought to the process (Creswell, 2013; Sin, 2010). Before starting this paper I wrote an autobiography that detailed 
my history and concern with this phenomena and why I decided to study virtual labs. I also kept a journal along with 
my field notes and interviews that commented on my thoughts and reflections on the interviews and observations. 
This helped me to clarify the conceptions of the students and ensure that I had accuracy in my interpretation of their 
meaning. 


557 



Data Findings and Connections 


The driving force behind the interview questions was to see how constructivist the class environment was 
through the eyes of the students and the strategies that students used to attempt and understand the content. By 
constructivist I mean how much the students were expected to form their own knowledge, how metacognitive and 
reflective they viewed their learning, and how authentic and interactive they saw the virtual labs. The community of 
inquiry framework developed earlier helps to shape the understandings of the student into varied conceptual levels. 
The other focus of the data analysis was on the strategies students used to understand the content. As recommended 
by Creswell (2013) and Schramm (2006) the focus in the interviews was on in-depth information from each 
participant to form rich and meaningful operations. 

The final phenomenographic categories formed “are treated as the final description and not part of the 
whole or essential description of the phenomenon” (Sjostrom & Dahlgreen, 2002, p. 341). The categories are 
arbitrary and imposed on the system by the researchers to help understand the phenomenon in detail (Sjostrom & 
Dahlgreen, 2002). The students’ alignment with one category or another was often dependent on the lab they were 
conducting. If the student had an interest in the lab they were more likely to view it in a positive light which aligns 
with the three approaches to interest research which includes the characteristics of the learner, the characteristics of 
the context, and the situational interest within the person (Renninger, Hidi, & Krapp, 1992). The phenomenographic 
categories formed are as follows: 

a. The general preference for physical labs over virtual labs and the blocking aspect this has on learning 

All of the students showed a preference for physical labs over the virtual labs and this is in contrast to many 
previous studies which showed no preference or advantage for virtual labs (Klahr, Triona & Williams, 2006; Pyatt & 
Sims, 2011). The excerpt below is from a dual interview with Cole and Samantha, students who were together at the 
remote site away from their science teacher. 

Victoria Raish: so how are these labs similar to the hands on labs you have done 

Victoria Raish: and how would you say they are different 

Samantha: one is real life stuff and one is on the computer 

Victoria Raish: why would you say the one on the computer is not real life stuff 

Victora Raish: and cole how would you say these virtual labs are similar to the hands on labs you did 

before 

Samantha: cuz its all like non touchable 

Victoria Raish: okay so do you think it loses some of the meaning because it is all virtual 
Samantha: yes 

Cole: you can interact with the labs but you can't with the vitual labs 
Victoria Raish: so why would you say you can't interact with the virtual labs 

Samantha: cuz u cant get hands on with the stuff on the computer (AdobeConnect Room, 2013-04-16). 

Emily, a student who was at the same site with the teacher, but due to scheduling conflicts was actually 
enrolled in biology in a room full of students taking chemistry shares this block on learning. 

Victoria Raish: so that is the one you did not like 
Emily: yes 

Victoria Raish: why not? 

Emily: i hated them 

Emily: and because they are so hard to learn and just do in a lab on the computer 
Victoria Raish: okay 

Victoria Raish: do you think doing the lab virtually lost some of the understanding 
Emily: yes, because i am actually a hands on person (AdobeConnect Room, 2013-04-23) 

These students do not feel that the virtual labs are interactive and can be manipulated in the same way that 
a physical lab could. These findings seem to contradict an earlier study done by Pyatt and Sims (2011) that show 
that “the instructional medium (physical or virtual), may have little or no effect on the learner’s ability to describe 
casual relations in inquiry settings” (p. 134). The conception shows that the students do in fact feel that the 
instructional medium affects their understanding of the content. 


558 



b. The preference for physical labs over virtual labs with the ability to see the value in virtual labs 

Engagement in authentic tasks is important for student learning which helps the learning process by 
investing students more in their academic work. A community of inquiry includes components of culture and 
enhancing understanding. Within the social presence there is affective expression and the culture of the environment 
which relates to the setting climate of the online classroom (Garrison & Arbaugh, 2007). Anna is clearly affected by 
this affective component of the lab, but does not let it hinder her learning experience of the lab. The following 
conversation comes from a prompt asking her what her favorite lab was. 

Anna: Probably the one with the bacteria slides because I've done a hands on lab which I felt was more 

exciting to see the real images under a real microscope. 

Victoria: So do you feel like doing it that way kind of lost some of the engagement of the lab. 

Victoria: do you think doing that lab virtually lost some of the learning experience 

Anna: I learned the same things but I guess I would say that the handson lab was more exciting. 

(AdobeConnect Room, 2013-04-12) 

The relevance of the virtual lab to prior knowledge and authentic learning experiences impacts the 
engagement of the students with the academic task. An excerpt follows in which Emily is discussing her favorite lab 
in biology and why she enjoyed it. The lab she is referencing involves the phenotypes and genotypes in a genetics 
unit. 


Victoria Raish: why did you like it 

Emily: i likd it because you are crossing them and i actually did like my fake baby type thing on a 
worksheet 

Victoria Raish: okay. ..you liked to see what they would look like? 

Emily: to see what color the eyes are, the hair, and like the thumb etc... and yes a lot (AdobeConnect 
Room, 2013-04-23) 

Learning in school is differentiated from learning outside of school by the fact that learning in school is not 
reliant on tools for help in understanding the content (Resnick, 1987). However, one of the students in this study 
found an advantage to the online classroom that allows tools for understanding. Anna is referencing being able to 
use the lab as a guide to understand the content of the class. 

Victoria Raish: so when he asks you a question in class outside of the lab you go back into the lab to look at 
it 

Anna: Yes we can always refer back to the lab. 

Victoria Raish: that is super helpful to always have it there 
Anna: Yeah it's a lot more helpful. 

Victoria Raish: do you think it is more helpful in a virtual lab because you can always go back to a concept 
that you are working on in class 

Anna: The fact that I can always go back to the lab is really helpful if I'm preparing for a quiz or test. 

Anna: Yes or you can look at it the other way and say that you can always go back to it at a later time. So 
time really isn't much of a factor with virtual labs. (AdobeConnect Room, 2013-04-12). 

c. Communication and sense-making via class mechanisms 

The community of inquiry emphasizes that in the absence of direct contact with the teacher, participants 
must attempt to “recreate the social and knowledge building processes that occur via moment by moment 
integration” (Shea et al., 2010). Communication is recognized as an important goal in science and one that in this 
context necessarily involves using the computer. In the following excerpts, Emily and Anna conceptualize using 
communication as the class platform was designed for. Emily is in a different class trying to complete the labs and 
shows the communication roadblock she encountered: 
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Victoria Raish: just because of having to do everything with the computer? 

Emily: well that and if they are the situation i am in, say if they have bio but have to be in a chem class, the 
teacher may not have the time t help them and i mean i had trouble the first couple times on them some of 
the labs are hard being new (AdobeConnect Room, 2013-04-23). 

Anna discusses being more comfortable participating in the virtual lab due to the design of the 
classroom. She reflected that she participated more in this setting than in her previous physical labs. The chat feature 
e of the class was heavily used to ask questions and that is the platform she is referencing. 

Victoria Raish: why do you think you are more comfortable doing it that way 

Anna: Since the class sizes are smaller mr. smith is able to answer all our questions right away rather than 
being to busy to answer of forgetting about it. (AdobeConnect Room, 2013-04-12). 

d. Communication and sense-making using the internet and other information resources 

In an online courseroom, the students have access to the tools that are not typical for those in traditional 
brick-and-mortar schools (Collins & Halverson, 2009). These tools take some of the control of the classroom away 
from the teacher because they are able to use the computer to find information. 21st century skills highlight the need 
to use resources effectively and engage in ICT literacy (Griffin, Bui, & Care, 2013). The following excerpts 
represent information retrieval strategies and communication techniques that were not necessarily designed to be a 
part of the course. Samantha and Cole recognize using Google as a strategy for learning about the content. 

Victoria Raish: how would you say you communicate while doing the labs 
cole: we dont.we just do them,thats it. 

Victoria Raish: what do you consider googling for answers 
Samantha: that depends on the question 

Victoria Raish: so if you have a question how do you get it answered 
Samantha: google 

cole: figured it out on your own or google it 

Victoria Raish: okay so you both have googled it before (AdobeConnect Room, 2013-04-16). 

Anna recognized that she does not use this strategy but realized the potential value of searching for the 
information. The excerpt followed a discussion of communication and how she finds information for the labs. 

Victoria Raish: do you think you would be comfortable with doing that on your own - using a search engine 
and finding a web page 

Anna: Yeah I think google would be a great alternate resource. (AdobeConnect Room, 2013-04-12). 

However, there is also a problem with students trying to think outside of the system and design constraints 
in the number of websites they can access. A recurring theme was the inaccessibility of many websites: 

Victoria Raish: and do you ever get blocked from sites you are trying to look at 
Victoria Raish: from the school 
Emily: yes 

Victoria Raish: because it says it is a game 
Emily: all the time 
Emily: i hate it 

Victoria Raish: yea.. .do you think that frustrates 

Victoria Raish: and delays your learnring 

Emily: it reallu does (AdobeConnect Room, 2013-04-23). 

The above quote referring to the blocking of websites was influenced by my personal knowledge of the 
school. I experienced this problem myself as a teacher and felt the frustration of the students in being unable to 
access quality sites that provided knowledge or simulations. 
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Discussion 


The data discussed in this paper do not represent the experience of all of the students at Tullahoma or 
blended schools in general. A clear theme underlying all of the interviews is that the previous conceptions and 
attitudes that students have toward their formal learning clearly affect their experience with the virtual labs. 

To ensure trustworthiness in a phenomenographic study it is important that the participants’ conceptions 
are asking follow-up questions to initial answers and addressing the topic later in the interview for clarification of 
meaning (Sin, 2010). However, the following context specific answers are not considered generalizable in this small 
scale study. It is proposed by Larsson (2009) that phenomenographic research can be generalizable if the sample 
size is large enough and the data analysis stage results in rich and deep information; however this view is not the 
most ideal for qualitative research. If the sample size is adequate there will be saturation in the variety of 
conceptions experienced by individuals with the phenomena in question. A limitation of this study is that the sample 
size is small (n=4) and thus the wide variety of conceptions has likely not been achieved so this study is context- 
specific. Tullahoma school is also unique in its environment and structure due to the hybrid nature and heavy 
involvement of the computer. The fact that some students are physically present in the classroom, but still interact 
with their teacher through the computer and can also have physical conversations with some peers and not others 
makes the likelihood of generalizable findings smaller. Due to the inexperience of the interviewer, some of the 
interview questions were leading to address the point I wanted to make in my research questions. An example of a 
leading interview question is shown below: 

Victoria: Does he just give you the answers or will he help you develop understanding by questioning you 

to help you develop the right answer? 

Emily: he does not tell me the answer, he helps develope understanding so that i do understand in the future 

(AdobeConnect Room, 2013-04-23). 

In contrast to what has previously been shown that virtual labs can still be considered interactive and that 
the models and simulations help to promote an inquiry based, constructivist centered approach by doing (Pyatt & 
Sims, 2011). The view held by Klahr, Triona, and Williams (2007) that “children's hands remain active and in 
control of the investigations” (p. 185) was not held by any of the students because they did not view it as the same 
activity as physical manipulation of materials. 

Through this pilot study, the need for an expanded community of inquiry for students in K-12 online 
learning is necessary. The original community of inquiry framework focused on higher education online learning. 
While most of the categories are extremely relevant to both situations, from a situative and activity perspective the 
context and environment necessarily influences the knowledge and learning that occurs and the difference in student 
characteristics and design of the learning environment should be explore through this framework to evaluate its 
applicability to the K-12 settings (Barab & Duffy, 2012). In future studies, a think-aloud protocol for the interviews 
will allow access into the thought processes while people are completing a certain task should be considered as the 
complexities of the interactions between the individual and their environment shape the experience of the students 
and can provide a different perspective than retroactive interviewing. 

Information literacy is a skill that is considered to be “pivotal in the pursuit for lifelong learning and central 
to achieving both personal empowerment and economic development” (Bruce, 2004, p. 8). Trefil (2006) sees 
scientific literacy as important to understanding major issues that are going on in the world. Virtual labs allow 
students to engage in more self-directed learning and earn practice with multiple 21st century skills through the idea 
of Just-in time teaching in which students only retrieved the information they need at the time they need it. Jonassen 
(2013) substantiates the idea by proposing that if students do not have a readily applicable context to apply the 
knowledge they just learned then it is not meaningful and increases isolated knowledge. Emily sees the main point 
of doing the virtual labs as helping with computer skills because of the type of school that they attend. Anna 
mentions that she uses the virtual labs to go back and review information that is discussed in class. This is something 
that is not possible in a physical lab where the lab is a singular classroom experience and cannot be revisited. Even 
Cole and Michelle, who were generally critical of the virtual labs, acknowledged that they use Google as a resource 
and this is a strategy the students are using on their own. If this strategy becomes acknowledge by the formal school 
environment, than the students can be scaffolded to find and evaluate information appropriately (ACRL Standards, 
2000; Garrison & Arbaugh, 2007). For this to occur, the school rules blocking certain websites needs to be revised 
so students can be self-directed learners and find access to reliable websites that provide important information. 

This pilot study has provided a few key insights into students completing their virtual labs in a blended 
charter school. Even though there is a significant amount of research on virtual labs, none have taken place within 
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the context of a school where students complete the majority of their curriculum via the computer. A future study 
would include a larger sample size that would fulfill the saturation of data and maximum variation in conceptions. 
Ideally, the software being used would also be analyzed for its instructional design principles and focus on a 
constructivist-based design to help understand the student experience with a particular online lab simulation. 
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Abstract 

Recent growth in use of project based learning in K-12 raises questions that warrant expanded research 
efforts. This paper shares results of a survey conducted to assess how researchers are keeping up with PBL use in 
schools, what topics are being addressed, and to identify opportunities for collaborative investigation. Results are 
based on unique features of PBL identified by participants, as well as common features and concerns. This paper 
also discusses efforts to create an interactive Google Site designed to serve as a shared resource that enables people 
to update or contribute to this work in the future. 

Objectives or purposes 

The purpose of our survey was to assess how researchers are keeping up with PBL use in K-12, what topics 
are being addressed, and to identify opportunities for collaboration or synergy among K-12 PBL researchers. 

Project based learning (PBL) has become a prevalent instructional innovation within the progressive K-12 reform 
movement, particularly among reform networks and smaller high school initiatives (American Institutes for 
Research & SRI International, 2004, p. 65). In recent years, schools, districts, and even states have been 
experimenting with substantial PBL initiatives (West Virginia Department of Education, 2013; Williamson, 2008). 
In Indiana, over 1000 teachers have received professional development (Gillenwaters, 2009; Staff Reports, 2011; 
University of Indianapolis, 2010; Indiana University School of Education-Indianapolis, 2010; 2012). Moreover, the 
UTEACH program, a nationally prominent pre-service STEM program has a full course about project-based 
instruction (Petrosino, 2012) and there is also significant interest in PBL outside the US (Problem Based Education 
SIG, 2012; Republic Polytechnic, 2007). The growth in PBL raises questions that warrant expanded research 
efforts. Educators, policymakers, and school community members may benefit from studies that provide 
information about such topics as effective classroom practices, leadership and school culture that supports PBL, 
professional development, as well as student achievement and engagement. 

PBL research efforts could be bolstered by a cohesive research agenda and a collaborative approach to 
overcoming methodological dilemmas. As an initial step toward connecting PBL researchers to this end, we 
developed and deployed an online survey to identify who is conducting PBL research, what their research interests 
are, and how they might wish to proceed in a joint research effort. We are currently seeking funding to convene 
interested parties in an opportunity to review and discuss the survey results as well as the final paper. Additionally, 
we have developed a Google Site to share the results of this survey and to enable people to update or contribute their 
own work and perspectives. There is a shared discussion board and spreadsheet of participants and resources, as 
well as shared documents that can be used virtually or to enhance face-to-face meetings. 


Perspective(s) or theoretical framework 

A major emphasis of PBL research has been problem-based learning as applied in medical, industry and 
higher education settings, but there is growing awareness of use in K-12 settings (Walker & Leary, 2009). To some 
extent research on PBL in K-12 may be hampered by lack of investment compared to what is often available for 
research in higher education and industry. What limited funding is available often goes into development and 
delivery of resources, not research per se. It is important to increase awareness of valuable research on PBL in K- 
12, and how research can better inform practices and policies. Providing scaffolds for collaboration and bringing 
researchers together, in person and online, may help to promote knowledge and models for advancing practices and 
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policies, adding value and helping the field to avoid implementation and research pitfalls. 

PBL in K-12 settings can vary by grade, subject and teacher, but it is understood to be learner-centered, 
constructivist-based instructional approach that is designed to support deeper, more engaged learning. This 
approach uses “projects” as vehicles to encourage student motivation contextualize content and concepts, and to 
provide a means for exhibiting and explaining what students have learned. Although there are subtle differences, 
PBL has much in common with problem-based or inquiry-based instruction (Barron & Darling-Hammond, 2008; 
Savery, 2006). All of these approaches attempt to promote academic rigor while promoting “soft skills” such as 
critical thinking, communication and collaboration (e.g., Trilling & Hood, 1999). They often encourage students to 
be responsible and resourceful for their own learning, to solve open-ended problems, and usually to create and 
present artifacts (e.g., as analyzed by Mitchell, et ah, 2005) as demonstrations of their learning. 

Methods, techniques, or modes of inquiry 

In the spring of 2012 the authors wrote and distributed the PBL in K-12 Research Survey (Ravitz & 

English, 2012) via SurveyMonkey. This method allowed us to gather information from distant individuals with 
minimal disruption and to follow-up when needed. We searched Google Scholar (scholar.google.com), Google 
(google.com) and our personal fdes and communications for researchers known to be working on various aspects of 
PBL. This included people who had published about PBL related to leadership, whole school reform, performance 
assessment, technology integration, collaborative learning and teacher professional development. We included 
academics, research-oriented school leaders, evaluation consultants, and informal learning researchers (e.g., Kanter, 
2012), as well as a small group of international scholars. We started with a list of 35 known researchers and asked 
them to complete a survey in which they would confirm their interest in PBL in K-12, provide information about 
their work, and recommend others. This eventually led to 81 people in total who were asked to complete the survey. 

Data sources 

We were able to obtain 58 responses (a 72% response rate) from across 19 states and 5 countries outside 
the US. Of these, 48 confirmed they are conducting PBL-related research and are interested in participating in or 
contributing to a collaborative effort in some way Respondents expressed interest in working on topics across a wide 
range of subjects and grade levels— 7 in all subjects and grades (including pre-service); 7 in secondary all subjects; 5 
in math in all grades, 12 focused on science in various grades, 6 on secondary social studies; 3 STEM across grades, 
and 2 elementary, self-contained. A majority of these were at university institutions, but some worked for non- 
profits, charter management organizations, or museums. Although we were disappointed with some who did not 
respond, this sample was considered fairly representative of the PBL in K-12 research community. Appendix A 
provides a list of potentially interested participants who answered the survey or were recommended by others. 

Results 


Interest in specific issues 

Researchers were asked to indicate their level of interest (on a 4-point scale) in 10 specific PBL issues, 
which included: pre-service and in-service professional development, designing, classroom management, teacher 
and student perceptions, accountability and other outcomes and technology integration. The average level of interest 
for all 10 issues was 3.2, indicating a high level of interest in all of these issues. Other interests identified by 
participants using open-ended responses included the design and use of PBL in informal environments, online 
professional teacher networks, cross-context research on PBL as a pedagogical culture, and the relationship between 
PBL and self-regulated learning. The study methods reported by participants included experimental, survey, 
qualitative and mixed methods. 
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Figure 1. Wordle of Unique Qualities of PBL 
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The above “Wordle” (Figure 1) is based on responses to an open-ended survey question asking participants 
to identify any unique qualities of PBL or terms that define PBL for them and distinguish it from approaches that 
may be used by others. These represent potentially important variations in PBL use that might be explored further. 


Common features of PBL or concerns 

Another open-ended question that proved to be fruitful asked respondents to describe common features of 
or concerns about PBL that might help unite research efforts. We analyzed these responses in order to identify 
major threads of interest. The specific threads identified include barriers to PBL use, clarifying what PBL is and 
how it works, identifying best practices to enable quality experiences, ensuring fidelity of implementation, student 
skills and benefits, and research concerns. 


The thread with the greatest number of responses (27) was identifying PBL best practices. Some of the 
practices noted included: 

• Generating of quality problems and projects 

• Promoting meta-cognition, self-regulation and self-direction 

• Creating authentic learning environments 

• Improving outcomes for diverse learners 

• Creating effective rubrics 

The thread with the second highest number of responses (13) was centered on barriers to PBL use. Some of the 
barriers identified included: 

• Addressing a standardized curriculum with PBL 

• Developing teacher skills for implementing PBL 

• Answering teacher concerns (classroom time, planning time, ability of students to self-regulate) 

• Integrating PBL within a culture of high-stakes testing and accountability 

• Identifying school conditions that support learner-centered approaches 

The third most commonly cited thread (11) was clarifying what PBL is and how it works. Example issues in 
this category included: 

• Developing a common definition or definitions of PBL 

• Describing how to develop driving questions to envelop curriculum standards 

• Developing models that enable us to represent teaching and learning processes 

• Dispelling the myth that PBL is “fluff’ 

The fourth greatest number of responses (6) related to student skills and benefits. Sample concerns included: 

• Measuring student efficacy and habits of mind developed through PBL 

• Clarifying the relationship between PBL and self-regulated learning or self-directed learning 

• Identifying ways students can benefit from real-world connections and collaboration 

• Demonstrating that there can be transfer of problem-solving skills 
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• Leveraging intrinsic interest and real-world contexts 

• Determining whether (or how) PBL can work equally well for all students 

Other threads related to measurement concerns, such as measuring student outcomes beyond test scores, 
measuring the quality of implementation and measuring fidelity to the intended curriculum were also noted. 

Interest in collaborating 

The survey asked participants to indicate their level of interest (on a 4-point scale) in seven types of 
interactions — including joint research projects, seeing work of others, sharing work, posting work in online forum, 
ongoing dialogue, live chats, and publishing opportunities. The average level of interest for all types of participation 
was 3.2, indicating a high level of interest in all types of participation. 

There was enthusiasm for sharing knowledge and finding opportunities to share knowledge and collaborate. 
A substantial proportion (40%) said it was “highly likely” local colleagues would benefit from a regional convening 
on PBL, while (47%) said it was somewhat likely and very few (12%) said it was not likely. Concerning a one-day 
national meeting (Figure 2), a substantial majority (67%) said it was “highly likely” they would attend such a 
conference on PBL in K-12, if funding was available to cover costs and it was linked to a national conference they 
were already attending. 


Figure 2. Participation likelihood for all day meeting, 
if funded and coinciding with a major conference 

Highly likely 

Somewhat likely 

Not likely 

0 10 20 30 40 




Tools for collaboration 

We asked about online collaborative environments that might be used to support collaboration among PBL 
in K-12 researchers. As shown in Figure 3, of the four options we offered in the survey GoogleDocs had an 
apparent edge in number of users, while Ning was at a decided disadvantage. Facebook appeared to have a slight 
edge in use on Wiki. There were also a large number of alternative technologies listed as an open-ended response 
by participants. These included EdModo, Skype, DropBox, Moodle, and others. 
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Figure 3. Use of collaboration tools 

To help us identify the most appropriate tools for collaboration, please 
indicate your current level of usage of each of the following. 



As a result of these data, we created a Google Site (Ravitz & English, 2013). This is linked to a group 
discussion board (PBLinK12research), and the group email address is used to share files or links, so that anyone 
with a Gmail account who is added to the group is automatically able to access shared documents and spreadsheets 
of participants and resources. We have not yet determined the best way to provide access to those without Gmail. 

As of now, access to the Google Site is limited to current participants. We are working on guidelines for 
participation and for adding new members. The plan is to add new researchers in waves, as we work to clarify our 
policies, especially how much of the information and discussion should be made public. To nominate yourself or a 
colleague for inclusion in the future, please contact the authors, or a colleague who identifies themselves to you as a 
participant. 


Scientific or scholarly significance of the study or work 

One significant result is that best practices emerged as the most frequent topic of discussion. Another of 
the four major threads that emerged was defining PBL. This may indicates that, in some cases, creation of the PBL 
model is still a primary concern, rather than what it takes to implement that model or its impact on students. We 
anticipate there can be productive discussion of best practices regarding specific components of PBL that are worthy 
of discussion (e.g., creating rubrics to assess specific outcomes), even while people may disagree or be agnostic 
about whether these or other specific components are critical to the definition of PBL. 

Two other major issues that emerged were barriers to implementation and impacts on students. These may 
represent a natural evolution of questions being asked, perhaps mirroring stages of concern research (Hall, 1979) for 
individuals. That is, until one has defined PBL and its practices it may be difficult or premature to conduct studies 
focused on implementation concerns (e.g., Ertmer & Simons, 2006; Pedersen & Liu, 2003) or student impacts (e.g., 
Grant, 2011; Peck, Peck, Sentz & Zasa, 1998; Vega, 2012; Vlereborne, 2010). On the other hand, it may be 
impossible to ignore questions about implementation and impacts even while one is defining and designing what one 
hopes will be effective and useable practices. 

In conclusion, the breadth and depth of perspectives offered in the survey responses we received provide 
opportunities to investigate theoretical and scholarly issues related to the above discussion and within each of the 
identified topic areas. In the future we hope to use the results of the survey and the Google Site to promote 
conversations among researchers who share interests on the above topics, or topics like teacher preparation, school 
leadership, use of technology, assessment and international implementation of PBL. As we work to build out the 
online web site we hope that we can begin to collect examples of research and recommendations for research in each 
of the above areas. 
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Appendix A: Geographic list of PBL in K-12 researchers 


State/Region 

# 

Institutional Counts (* indicates interest in hosting local meetings) 

Indiana 

30 

Purdue (10) * 

University of Indianapolis-CELL (12) * 

Indiana University-Bloomington (4 )* 

Indiana University-Purdue University Indianapolis (4) 
Indiana University-Purdue-Columbus (1) 

Israel 

10 

Weizmann Institute of Science (3) * 
Technion (2) 

Institute for Democratic Education (2) 
Tel Aviv University (2) 

Bar Ilan University (1) 

California 

7 

New Tech Network (2) 
Independent Consultant (1) 

Mills College (1) 

San Jose State University (1) 

SRI International (1) 

Buck Institute for Education (1) * 

Other International 

5 

Scotland - University of Edinburgh (2) 
Germany - Ruhr University Bochum (1) 
Cyprus - CARDET - University of Nicosia (1) 
Canada - University of Toronto/OISE (1) 

Minnesota 

2 

Hamline University (1) 

Minnesota State University-Mankato (1) 

New Hampshire 

2 

University of New Hampshire (2) 

North Carolina 

2 

North Carolina State University (2) 

New York 

2 

New York Hall of Science 
Teachers College -Columbia University 

Texas 

2 

New Tech Network (1) 
University of Texas at Austin (1) 

Virginia 

2 

George Mason University (2) 

Alabama 

1 

Auburn University 

Colorado 

1 

University of Colorado-Boulder 

Iowa 

1 

Iowa State University 

Michigan 

1 

University of Michigan 

New Jersey 

1 

Rutgers University 

New Mexico 

1 

University of New Mexico 

New York 

2 

New York Hall of Science 
Teachers College -Columbia University 

North Dakota 

1 

University of North Dakota 

Tennessee 

1 

University of Memphis 

Utah 

1 

Utah State University 

Washington 

1 

University of Washington 


Note. List represents survey respondents and people they said they could identify at the time of the survey 


572 




Design and Development of a MOOC: The Value of a Collaborative Process 


Stephanie Richter 

Faculty Development and Instructional Design Center 
Northern Illinois University 
DeKalb, IL 


Keywords: MOOC, instructional design 


Abstract 

Few current educational technology innovations have received the attention of Massive Open Online 
Courses (MOOCs). Both trade and popular publications have featured articles extolling the promise that MOOCs 
will revolutionize higher education. Most of the discourse, however, is limited to hypothetical suppositions. This 
paper describes the unique collaborative process one institution followed to design and develop their first MOOC, 
including recommendations for other institutions beginning their MOOC journey. 

Overview 

Massive Open Online Courses (MOOCs) are not new, but in the past year, they have roared into the media 
spotlight. Some argue that MOOCs will revolutionize higher education (Cormier & Siemens, 2010; Pappano, 2012), 
diminishing the role of universities, and possibly even eliminating them altogether (Kolowich, 2013; Parry, 2013; 
Shirky, 2012). Others laud MOOCs for bringing education to those who would not otherwise have access to higher 
education by increasing access and decreasing cost (Coursera, n.d.; Roller, 2012). 

MOOCs have suffered criticism, too. Critics argue that the majority of MOOCs may be ineffective, or at 
least that there is not enough research to determine that they are effective (Marucs, 2013). They point out that many 
MOOCs utilize primarily broadcast media like video lectures and offer little (or no) interaction between the 
instructor and students. In addition, despite enrollments in the thousands, and frequently over 100,000, completion 
rates for MOOCs tend to be very low (Jordan, 2013). 

Gaps in the Literature 

Despite the broad hopes of MOOC proponents, very little is known about learning and student success in 
MOOCs. There is a distinct lack of research on learning outcomes in MOOCs. Most of the discourse on MOOCs 
focuses on the platforms (especially Coursera, Udacity, and EdX) and the promise of radical change to higher 
education. There is a need for more discussion of course design, development, assessment, and evaluation of 
MOOCs to determine the impact they will have on students, higher education establishments, and society. 

For example, who enrolls in MOOCs? Are MOOCs reaching their target audience - those individuals who 
could not otherwise attend higher education? Is there evidence that students learn as a result of a MOOC? How do 
design choices affect student motivation, completion, and learning? Before considering MOOCs and a DIY 
credential as viable replacements for traditional degrees, these questions must be answered. 

On the other hand, for institutions considering offering their first MOOC, there are even more practical 
considerations. How much time does it take to design and develop a MOOC? How much financial investment is 
necessary? What technologies and proficiencies should be considered? How can effective online pedagogies be 
scaled for massive audiences? 

Given the lack of literature, it seems many institutions have been reluctant to share their experience with 
design, development, and delivery of MOOCs, which makes it challenging for others to determine effective practice. 
It is through such sharing that the community at large can learn and improve, and is the spirit in which this paper is 
written. 


Project Information 

In 2012, a team at Northern Illinois University (NIU), a Midwestern public university, began designing the 
first MOOC to be offered by the university. The MOOC was the project of Dr. Greg Long, a professor in the School 
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of Allied Health and Communicative Disorders, who teaches a face-to-face course at NIU titled “Disability in 
Society.” Long developed the course 10 years before because, while disability is a fact of life for millions of people 
in the world, it tends to make people uncomfortable, so it is not talked about. Since that time, the course has become 
quite popular on campus as a general education course, with over 300 students enrolled every semester. Disability in 
Society provides a broad overview of disability awareness, background knowledge, and selected issues of disability 
across the lifespan. 

According to Long, every semester, students report that they view disability (and individuals with 
disabilities) more positively as a result of the course. Students begin to see accessibility as an issue of equal access 
and societal. Student feedback from the course indicated that the course was very beneficial to the students who took 
it, and several students every semester report that the course should available to more people. This encouraged Long 
to develop a MOOC, so that the information he shared in the course could be freely available to a wider audience. 

The MOOC, titled “Perspectives on Disability,” addresses the history, culture, and stigma of disability, 
with a goal of raising awareness and encouraging dialogue about issues related to disability. Long felt it was 
important for the course to be appropriate for anyone from middle school and up, with or without a background in 
disability. His first step was to begin recruiting assistance from other faculty, staff, and students from across the 
university. 


Collaborative Design Approach 

From the beginning, the MOOC was a collaborative project. Unlike other institutions, NIU did not have a 
formal structure in place to support the design and development of a MOOC, so the collaboration grew throughout 
the project. Long’s first contact was to the Faculty Development and Instructional Design Center on campus for 
assistance. He recruited over 50 graduate and undergraduate students to assist with the design process and to provide 
feedback on the content of the MOOC. He also partnered with a professor of documentary filmmaking and began a 
project with her students to capture and edit the interviews. With additional support from technical services, media 
services, and the outreach center, the project quickly became a collaborative campus-wide project. 

Throughout the project, Long served as the subject matter expert for course design and content. He 
provided expertise on what content to include and how to present it, including developing the initial draft 
storyboards, selecting readings, and identifying additional resources, for each lesson. Long was the face of the 
course and recorded all of the lectures. When the MOOC began, he also facilitated the course delivery by posting 
materials, sending announcements, and responding to student concerns and questions. 

Long consulted with the Faculty Development and Instructional Design Center for advice on delivering a 
MOOC and recommendations on the use of technology. Generally, the Center supports effective teaching practices 
and technology integration for NIU’s faculty, academic support staff, and graduate teaching assistants. For this 
project, the Center provided guidance about the pedagogy of teaching a MOOC, developing materials for online 
delivery, and assessing a massive audience. In addition, the Center established a relationship with Blackboard 
CourseSites to deliver the MOOC and forged partnerships with other units within NIU, such as the Division of 
Communications and Marketing (formerly the Division of University Relations), Media Services, and the Office of 
General Counsel. The Center continued to be engaged during the delivery of the course by offering feedback on the 
course and assisting with the deployment of material and assessments in the course management system. 

A defining characteristic of the collaborative approach for designing Perspectives on Disability was a 
commitment to involving students. The goal of doing so was to provide the students with a voice in the course 
design, to promote quality through increased feedback, and to provide experiential learning opportunities for NIU 
students. Overall, more than 50 graduate and undergraduate students contributed to the course design and 
development. The largest group met as part of an independent study in the spring 2013 semester. These students 
learned about online course design, helped to select the weekly topics, drafted initial outlines for the lessons, and 
researched potential readings for the MOOC. The seminar students also reflected on their experience with taking the 
traditional course and unanimously agreed that some of the strongest aspects of the course were the guest speakers 
who presented about their experience with disability. Because of this, including those stories became a high priority 
for the course. 

In the same semester, another group of students collaborated to create disability tutorials for the 
Presidential Commission on Persons with Disabilities. The videos were a collaboration between students with 
disabilities or family members with disabilities and documentary film students from the Department of 
Communication. While these videos were created to accompany a text-based tutorial, some of the videos are 
included in the lessons of the MOOC. In addition to these tutorials, two graduate students from the Department of 
Communication recorded all of the lectures and interviews with the guest speakers. 
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Two graduate students from the Department of Educational Technology, Research, and Assessment served 
as interns for the course development and delivery. One developed the lecture videos by combining the lecture =and 
guest speaker videos with visuals. This intern also compiled transcripts for the lessons and posted the videos on 
YouTube as well as in the course. The second intern wrote the quizzes and built them in the course management 
system. She also monitored interaction on the discussion boards and replied if necessary or forwarded them to the 
professor for follow-up. 

In addition to students, a number of other faculty and units at NIU were involved in the collaboration. Dr. 
Laura Vazquez, Professor in the Department of Communication, supervised the students who recorded the video 
tutorials, guest speakers, and lectures. The Division of Communications and Marketing provided advice for 
marketing and promoting the MOOC, including writing and distributing several press releases about the course. The 
Division also used the official university social media channels to spread the stories. Media Services created the 
graphics, including course logos and social media profile images. They also created caption files for the lessons. The 
Office of General Counsel provided perspective on the legal risks and concerns associated with offering the MOOC 
and developed the terms of use for the MOOC with the Faculty Development and Instructional Design Center. 

Results 

The resulting course is ten weeks in length. Each week covers a facet of disability, including models of 
disability, perceptions and definitions, portrayal in language and the media, education, community living, and 
employment. Each week’s lesson consists of four to six short video segments that are approximately eight to twelve 
minutes long. Based on feedback from the seminar students, each week (and most of the individual segments) 
include stories from the guest speaker interviews. 

Data on student success in the MOOC is not yet available, but the design and development process has 
been successful. The resulting course is driven by the stories and experiences of individuals with disabilities, so 
students connect to the human side of the content. Content is media-rich and uses a combination of lectures and 
interviews presented through videos and text-based reading assignments. The course is accessible to all students, 
however, regardless of prior knowledge, education level, or disability. 

This collaborative approach brought multiple departments and individuals together so that weaknesses or 
deficiencies of individual skill were not an issue. Seeking student input strengthened the design of the course. While 
there were obstacles and challenges throughout the process, maintaining a focus on sharing knowledge rather than 
gaining institutional or individual recognition inspired all of the project participants to maintain their commitment to 
the project and ensure the final product was the best possible. 

Recommendations 

This experience has led the team to develop a number of recommendations for others developing a MOOC. 
In the spirit of the collaborative approach to designing the MOOC, it is hoped that these recommendations will help 
others to develop stronger MOOCs and advance the discussion around pedagogy of MOOCs. 

The first recommendation is perhaps the easiest and most obvious: participate in a MOOC. Preferably, 
participate in several, on multiple platforms, led by different faculty. If possible, participate actively with a goal of 
completion, and reflect on the design and how it impacts motivation and persistence. The experience will help to 
guide the design and development process. For Perspectives on Disability, several staff of the Faculty Development 
and Instructional Design Center had completed several MOOCs and engaged in extensive research into the structure, 
pedagogy, and assessment of MOOCs. This experience was an important factor in the design of the MOOC. 

Unlike many other courses and teaching opportunities, it may be possible to have full control over the 
choice of topic for the MOOC. Choose a topic that you are passionate about, so that you are motivated to create the 
course and facilitate thousands of students. The entire collaborative team for Perspectives on Disability was 
committed to the topic of disability and this dedication helped the team maintain focus and motivation throughout 
the project. 

It is also important that the topic is appealing to MOOC students, particularly if the course will not result in 
college credit. Since the choice of topic may be the most important decision in the success of the MOOC, consider 
conducting a needs analysis or gather extensive feedback from colleagues or industry leaders to determine the 
importance of the topic for the intended audience. In the case of Perspectives on Disability, student feedback from 
the traditional version of the course provided evidence for the importance and applicability of the content. Further 
conversations with disability rights groups and K-12 educators reinforced the need for the topic. 
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Given the potential diversity of MOOC students, it is also important to determine the target audience for 
the MOOC, particularly education level and pre-requisite knowledge, skills, and abilities. This will impact the level 
of language used throughout the course, the depth and breadth of the content, and assessment approach. It is 
important to remember that, in many MOOCs, a substantial portion of the students are non-native English speakers, 
so it may be challenging for some to understand complex language or write lengthy assignments. The cultural aspect 
of diverse students is a factor when considering content, as well. Perspectives on Disability was designed for any 
student from middle school through adult and primarily for those without background in disability. One area that 
was challenging was incorporating perspectives from outside of the United States, so one assessment option was to 
submit information about the topic from a different cultural perspective. 

It is also strongly recommended to build a team as early as possible. This team should cover design and 
development, subject matter expertise, review and testing, technology support, media production, marketing and 
publicity, legal, and others as necessary. A team approach for delivery and facilitation of the course would be wise, 
as well. Individuals on the team may fulfill multiple roles. The team for Perspectives on Disability grew over time, 
as dictated by the progress of the project, which was both beneficial, since individuals were able to join and leave 
the project as needed, but also a challenge, since there was a delay between identifying a need and finding someone 
to fill the role. 

The final recommendation is to use MOOC design and development to think differently and creatively 
about what defines a course. Because they are detached from other institutional regulations about credit, assessment, 
and seat time, it is possible to design a MOOC to explore alternatives for content delivery, community engagement, 
and assessment. For example, in Perspectives on Disability, the content delivery approach was somewhat traditional, 
based primarily on lectures by a faculty member. The video format, however, allowed the guest speaker stories to 
weave with the lecture in a way that would be difficult in a traditional course. In addition, the course assessment 
strategy was unique, with a combination of objective assessments (quizzes) and the choice of multiple activities that 
were “graded” on submission only. These are just a few examples; there are a multitude of ways to experiment with 
course design that may be discouraged in a traditional course but are possible in a MOOC. 

Conclusion 

The opportunity to experiment by designing and delivering a MOOC has been a valuable experience for 
everyone involved. Using a collaborative approach to the design, development, and delivery has permitted a wide 
variety of individuals to benefit from the experience, including faculty, staff, and students. 
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Abstract 

A challenge of online learning is enhancing students' sense of social presence, that is, their perception of “being 
there” with others. At the University of Hawaii at Manoa College of Education, we are working on a research and 
development project with the goal of enhancing students' social presence through the use of a mixed-reality 3D 
virtual learning environment. Our project, the Holodeck@UH, attempts to bring the virtual world and the real world 
together as a mixed-reality “mash-up”. Using the Holodeck, on-campus and online students are able to come 
together both physically and virtually and take part in class and other collaborative activities. This article describes 
the initial phases of our design-based research effort in which we designed and built the Holodeck, from concept to 
advanced working prototype, and our initial evaluation efforts. Findings from an initial formative evaluation and 
usability study are presented and discussed. 


Introduction 

The topic of online learning conjures images of course management systems like Blackboard and Sakai, in 
which course materials are made available to students, students can submit their assignments to instructors and the 
class can discuss topics using discussion forums, text chat and in some cases audio and video conferencing. Given 
remarkable enrollments in online courses (Picciano & Seaman, 2008) and predictions that over 10% of all K-12 
classes will be offered online by 2013 (Christensen, Horn, & Johnson, 2008), it is clear that educators and 
educational institutions see enormous value in online learning. However, course management systems are but one 
medium to enable learning via the Internet. Other mediums such as multi-user virtual environments (MUVEs), 
virtual worlds and serious games are gaining traction as viable learning technologies in educational institutions 
(Dalgarno & Lee, 2010; Livingstone, Kemp, & Edgar, 2008; McLellan, 2004). Arguably, the affordances of these 
technologies can provide for more social learning than traditional CMS, which can be socially isolating 
(Mclnnerney & Roberts, 2004). Researchers are exploring a variety of MUVEs in order to learn how to best 
leverage the characteristics and capabilities of the medium to impact learning outcomes. For example, applications 
built using MUVE technology such as Quest Atlantis (Barab et at, 2011; Barab, Sadler, Heiselt, Hickey, & Zuiker, 
2007), River City (Dede, Clarke, Ketelhut, Nelson, & Bowman, 2005; Ketelhut, Nelson, Clarke, & Dede, 2010) and 
EcoMUVE (Metcalf, Dede, Grotzer, & Kamarainen, 2010) are showing the potential of MUVE technologies as 
teaching and learning tools. Given the promise of learning benefits such as enhanced motivation and engagement 


578 



attributable to media like three-dimensional virtual learning environments (3D VLE), MUVEs and serious games, 
interest is growing. 

The majority of educational MUVES focus on learners interacting within virtual worlds to complete 
learning activities and assume that all users will be interacting in the virtual world. However, the Holodeck@UH 
system we describe in this article allows students to be present both in the physical classroom and remotely in a 
virtual classroom. Our system is inspired by a prototype system developed by researchers at the University of Essex 
called MiRTLE (Mixed Reality Teaching and Learning Environment). In MiRTLE, a virtual world is connected 
with a real-world classroom to create the mixed-reality teaching and learning environment (Callaghan et ah, 2008; 
Gardner, Ganem-Gutierrez, Scott, Horan, & Callaghan, 2011; Gardner, Scott, & Horan, 2008; Horan, Gardner, & 
Scott, 2009). In pilot studies, researchers from the MiRTLE project found that virtual world (VW) and real-world 
(RW) students would naturally and spontaneously interact with one another, which caused the barriers of VW and 
RW to blur. Instructors reported that teaching in MiRTLE did not disrupt their classes or students’ learning. The 
implications of these findings were that using MiRTLE may increase the sense of presence for VW students, RW 
students, and the instructor (Gardner et al., 2008). 

The purpose of this article is to provide a case description of early efforts in the design, development, and 
evaluation of our mixed-reality learning environment at the University of Hawaii at Manoa. The project is guided 
by a design-based research (DBR) trajectory and uses a technology system that was constructed using completely 
Free and Open Source Software (FOSS). The project was inspired by local challenges in facilitating collaborative 
learning at a distance. Higher education students in the Pacific islands are geographically dispersed and, given the 
distances between the islands, often perform their studies through online distance education. Creating an online 
environment to enhance social interaction and community-building at-a-distance for these students can often be 
challenging. Currently, a number of online course delivery options are used to provide these students with quality 
instruction at a distance, such as course management websites and web conferencing tools. Our best practices in 
distance education and our culturally-responsive pedagogies acknowledge and respect that our students value being 
able to celebrate community and connect socially with classmates and teachers. However, for students whose 
cultures value social interaction and building relationships, traditional online learning tools can feel inadequate. We 
have designed the Holodeck@UH to help bridge this gap. Our system, is designed to meet a unique and practical 
need through the use of a mixed-reality teaching and learning modality that “mashes” a 3D virtual learning 
environment with a real-world classroom. The case description we provide here outlines the design and development 
of our initial functional prototype and reports the findings from a formative evaluation study we performed on the 
working prototype. 

The Holodeck@UH Mixed-reality System 

The Holodeck@UH is a “mashup” that combines a real world space (a physical classroom where students 
meet in-person) with a virtual world (a three-dimensional multi-user virtual environment where students meet as 
avatars). We use the term “mashup” to mean a hybrid environment that uses technology to creatively mix elements 
of the real world and virtual world to create a unified experience. By providing two-way blending of the real and 
virtual world, learners are able to interact in both real and virtual spaces in ways that may be more natural- feeling 
and promote social interaction. Figure 1 provides a general illustration of this two-way connectivity. 
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Figure 1. Top-down conceptual view of two-way connectivity between the virtual and real world. 

With the Holodeck@UH, we meaningfully blend online and face-to-face interaction and collaboration, 
providing students with a deep social experience. Our design and evaluation work has advanced our understanding 
of how to design virtual places and associated instruction so as to enhance online learning. At the core of our design 
is the idea that students should feel socially present with others, whether they are located in the real world (RW) or 
virtual world (VW). By “socially present” we mean that students have a feeling of “being there” with others 
(Biocca, Burgoon, Harms, & Stoner, 2001). According to Horan and colleagues (2009) virtual worlds can enhance 
the community and social interaction between its users and provide a greater sense of presence than traditional 
online collaboration and communciation technologies (instant messaging, chat, audio/video conferencing). Key to 
our design for enhanced social presence is creating high-fidelity representations of our students, whether they be 
attending class in the RW or VW. For our system, we combine immersive VW technology, voice over IP, high- 
speed networking and high-definition streaming video to create the mixed-reality experience. Local students attend 
class in a traditional classroom equipped with a high-definition projector, noise-canceling conference microphones, 
speakers, and a high-definition streaming web camera. The HD data projector is used to provide a view of the VW 
to those students who are physically present. From the perspective of the RW student, she enters the physical 
classroom, sits down, and is presented with a view of the VW on the projector screen at the front of the classroom. 
She is able to hear the avatars in the VW conversing and see them interacting with one another. The RW student 
may choose to log in to the VW and join the other students as her avatar, thus being present in both the RW and VW 
at the same time. Figure 2 provides a top-down schematic of our RW classroom. This represents one side of our 
mixed realities. 
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Figure 2. Top-Down Schematic of the Holodeck@UH real-world classroom. 

The VW classroom features three screens prominently displayed at the front of the virtual classroom. One 
screen displays a shared document that is used for collaborative note-taking by all students (real-world and virtual). 
Another screen displays the output from the high-definition streaming webcam, so that the students in the VW are 
able to view the RW classroom in real-time. The third screen displays the instructor's computer screen. Distance 
students attend class by logging in to a VW where they are represented as avatars, speak using microphone-equipped 
headsets, use avatar movements and gestures to interact, and interact with objects in the virtual-world using their 
mouse. From the perspective of the VW student, she logs in to the VW, navigates to the classroom area, and “parks” 
her avatar in one of the available seats. She is able to see a live streaming webcam video of his instructor from the 
webcam on his instructor’s computer. She is also able to see a HD video stream of the classroom and the students 
who are working there and is able to hear them speaking. Figure 3 provides an annotated screenshot of the VW with 
streaming webcam video of the physical classroom being displayed on a display. This represents the other side of 
our mixed-reality environment. 
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Figure 3. Screenshot of the VW displaying streaming video of the real-world classroom. 

Free and Open-Source Software and Design-based Research 

Free and Open Source Software is an approach to software development and distribution that includes 
source code and forms of licensing which permit ready customization and evolution while preserving the software as 
a common good. The benefits of using a FOSS VW platform are manifold. FOSS allows for ready customization 
and evolution to meet local needs, for iterations to meet special requirements of target users and for free access to 
source code. FOSS provides opportunities for designers, developers and users to participate in the community 
development effort that simultaneously contributes to meeting local needs (Carmichael & Flonour, 2002; Lin & Zini, 
2008). Open source software is particularly useful for educational application development in that it helps to 
establish a closer relationship between development communities, educators and users, so that the software can be 
iterated based on the needs and special requirements of the target users. The integration of the practices of teaching 
and learning with the flexibility and freedom to develop makes FOSS a suitable alternative to commercial software. 
Indeed, FOSS is gaining traction for its potential benefits over proprietary counterparts in the development of multi- 
user 3D VEs both for educational and enterprise applications. This interest is spurred by the flexibility, 
customizability and extensibility of FOSS 3D VE platforms such as Open Cobalt (http://www.opencobalt.org/), 
Open Qwaq (http://code.google.eom/p/openqwaq/), Open Wonderland (http://openwonderland.org/), OpenSim 
(http://opensimulator.org/) and others. For example, Young (2010) discusses the decision to use the NSF-sponsored 
FOSS platform Open Cobalt over the proprietary Second Life platform due to educators’ lack of control of the 
proprietary environment. In Kappe and Guetl (2009), the researchers discuss development of a virtual conference 
room for knowledge transfer and learning purposes and their preference for FOSS software toolkits due to the ability 
to customize and add functionality to such virtual worlds as well as their ability to interoperate with other virtual 
worlds, including Second Life. Zutshi and Sharma (2009) compared the usability and acceptability of two 
proprietary platforms, Second Life and Qwaq Forums, and one FOSS platform, realXtend, for collaboration within 
an enterprise. While the authors reported success with the realXtend platform, they were unable to achieve their 
goals with the proprietary platforms. The authors note the flexibility of realXtend enabled building an environment 
that users found more consistent with their real work situation than was possible with the other platforms. 
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The MiRTLE project used the free and open-source VW platform Open Wonderland 
(http://openwonderland.org) to create their mixed reality classroom. Open Wonderland is a collaborative 3D virtual 
worlds toolkit that allows for the creation of multi-user immersive VWs. Unlike other VW platforms like Second 
Life, which is an entertainment platform. Open Wonderland is built for deeply immersive collaboration and targeted 
at the education, business, and government sectors. However, Open Wonderland has limitations due to being a 
somewhat immature project. In our tests, we found it often to be unstable, to have poor performance with more than 
15 avatars, and to frequently break due to Java updates. Because we needed a system that was reliable and 
consistent, these problems in testing led us to another FOSS VW platform called Open Simulator (OpenSim). We 
chose OpenSim due to its vibrant FOSS community, stability, ability to host multiple logged-in avatars 
simultaneously, and potential familiarity to prior users of the SecondLife platform. We found OpenSim to be highly 
flexible and to allow for a great degree of customization at a deep level. Indeed, OpenSim has gained a reputation of 
being very useful to educators. While choosing this technology, we considered the arguments made by the designers 
and developers of the MiRTLE environment, who noted that OpenSim was based on C# and .Net technologies, and 
therefore was limited in terms of its cross-platform capabilities. Much had changed since the original development 
of MiRTLE, however. We found that OpenSim is able to run on Macintosh, Windows, and Linux systems, and even 
on Android. It is somewhat ironic that the very reason Open Wonderland was chosen for the MiRTLE project, that 
is, because it was based on Java, was the same reason we ultimately chose to move from that platform to OpenSim. 

While the benefits of the FOSS approach for development of 3D VLEs are compelling, FOSS software 
solutions bring with them unique challenges. For instance, while FOSS allows for profound flexibility, it can also 
result in difficulties due to great diversity in implementations and the need for highly knowledgeable local staff that 
have the capability to participate in broader FOSS communities (Laffey, Schmidt, & Amelung, 2010; Schmidt, 
Galyen, Laffey, Ding, & Wang, 2010). Other researchers’ (e.g., Kappe & Guetl, 2009; Young, 2010; Zutshi & 
Sharma, 2009) note the high requirements of hardware and professional knowledge of the personnel in the 
implementation. This challenge could well be the primary impediment to implementing FOSS 3D virtual 
environments. Nonetheless, FOSS as a development methodology brings many benefits to designers and developers 
of instructional systems, particularly when instructional designers need to agilely revise, adapt, make changes and 
re-implement designs to fit the target context and adapt to unexpected or emergent contextual variables. 

Instructional technology both constrains and affords learning, as the unique tools and capabilities presented by the 
technology allow for novel and innovative learning approaches, yet the limitations and tradeoffs that are part of 
technology implementations circumscribe the range of learning activities which can be performed. How often do 
instructors desire that a software suite could perform a certain desired function or that it were easier to access certain 
functionality? When required design improvements necessitate changes to underlying software, such changes are 
typically not possible unless the software licensing allows for inspection and modification of the underlying source 
code. While the majority of purchased, off-the-shelf software does not allow for this, FOSS does. 

The ability to enhance and evolve an instructional and learning system is of particular interest to 
researchers performing design-based research. DBR is theory-driven design, wherein the goal is not only the 
iteration of a product but also the advancement of a design theory for optimal learning and performance within a 
naturalistic context, usually in relation to the use of technology (Design-based Research Collective, 2003). In 
addition, DBR addresses specific, complex, and important educational problems (Reeves, 2006) by systematically 
testing designs in context with each implementation and analysis informing the next iteration of the design theory. It 
has been called an iterative cycle of design and enactment or implementation, followed by analysis of the 
implementation, theory iteration and redesign (Wang & Hannafin, 2005). These iterations of design and 
implementation have the goal of establishing the relevance of the implementation and ultimately its impact (Amiel 
& Reeves, 2008). FOSS seems to be a natural fit for DBR because it allows for maximum flexibility while iterating 
an instructional technology intervention due to its ability to be studied and manipulated at a very deep level, while at 
the same time promoting community and software for the common good. 

Methods & Findings 

Harnessing the power of advanced technologies so as to improve learning outcomes is complex and 
difficult. Researchers note that traditional research methodologies may be limited in their ability to connect 
technology-rich educational interventions with educational impact and point to design research (DBR) as an 
appropriate development and research methodology for technology-rich educational interventions that is focused on 
establishing their impact on real-world educational problems (Amiel & Reeves, 2008; McKenney & Reeves, 2012). 
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DBR systematically tests intervention designs in context, with each implementation and analysis phase iteratively 
informing proceeding phases (Wang & Hannafin, 2005). Reeves, Herrington and Oliver (2005) forward a series of 
principles for performing DBR which focuses on strong collaboration between researchers and practitioners and 
continual refinement of processes, questions and protocols. Because DBR embraces incomplete knowledge and 
allows for iterative expansion, promotes collaboration between researchers and practitioners and is well suited for 
technology-rich educational interventions, we have selected it as the methodological approach for our design, 
development and evaluation process. Following a DBR trajectory allows us to develop a robust product that is 
shaped by participant expertise, literature and especially usage testing. The design and research presented here 
represents the near-conclusion of one complete DBR cycle of analysis and exploration, design and construction, and 
evaluation and reflection. 

A formative evaluation and usability study was conducted in October of 2013. The purpose of the 
formative evaluation was to explore perceptions of the Holodeck@UH among educational technology students and 
professionals. The purpose of the usability study was to examine perceived ease of use among educational 
technology students. The ultimate goal of this research was to test our designs, identify areas for improvement, and 
advance design principles. Research questions that guided the study were: 

1 . How do participants perceive the mixed-reality blending of face-to-face and online courses? 

2. How do participants perceive social experience, sense of connection, and community in mixed-reality 

online and face-to-face modalities? 

3. How do participants perceive the ease-of-use of the Holodeck@UH system? 

An anonymous survey was developed using Google Forms, consisting of 20 questions, 15 of which were 
closed-choice and five of which were open-ended. The survey presented three short video vignettes of activities in 
the Holodeck@UH, after each of which participants were asked to answer questions based on what they had viewed. 
The survey was piloted with four educational technology professionals and amended based on feedback from the 
pilot. The research was also submitted to the campus institutional review board, which resulted in the research being 
given an exempt status. After piloting and IRB clearance a request to participate in the survey research was sent out 
to the University of Hawaii's educational technology list-serv, with a follow-up being sent one week later. After one 
week, 40 participants had completed the survey. Participants' reported their ages as 12% between 18 and 30, 37% 
between 31 and 40, 23% between 41 and 50, and 28% as 50 or greater. Reported occupations included 
teacher/instructor (44%), student (13%), technologist (23%), and other (20%). 

For the usability study, a think-aloud method and a usability protocol “script” were developed using 
methods forwarded by Krug (2010). Participants were recruited by asking educational technology instructors for 
recommendations of students they perceived to be either (a) very technically proficient or (b) somewhat technically 
proficient. This was so that we would be able to recruit a sample with maximum variation in terms of abilities. Of 
four volunteer participants, two participants were chosen, both males, one of whom reported his technical abilities to 
be “very high” and the other of whom reported his technical abilities to be “pretty low.” The usability study took 
approximately 45 minutes to complete, including answering post-test interview questions. Participants performed a 
number of tasks in the virtual environment such as performing navigational tasks with their avatars and locating 
information. Participant interactions were recorded using screen recording, webcam recording, and eye-tracking. 

A preliminary analysis of the evaluation and usability data was performed. Analysis of the evaluation data 
suggests that participants perceptions are generally quite positive, with participants describing the mixed-reality 
environment as innovative, engaging, and exciting. Participants report that using the Holodeck@UH could enhance 
the sense of community for students who are taking online classes on neighbor islands. However, participants also 
indicate that they would like to learn more about how the Holodeck@UH can impact learning, and how it would 
work in a real-world setting as opposed to video vignettes. Responses from non-demographic closed-choice 
questionnaire items suggest that participants find the mixed-reality modality to make the learning experience in the 
virtual world more personal, but agree that learning in mixed-reality appears to be complicated. Nonetheless, they 
agreed that mixed-reality seems like an engaging way to leam online and that our design could enhance a sense of 
community for students on neighbor islands. Nearly all participants agreed strongly that they would like to class in a 
mixed-reality environment like the Holodeck@UH ; however, they also agreed that a mixed-reality system like the 
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Holodeck@UH should not be used to replace other online course technologies, but instead should be used to 
compliment those other technologies. These findings are presented in Figure 4 below. 


Formative Evaluation Results (n=40) 
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Figure 4. Formative evaluation responses on closed-choice questionnaire items 


The usability study focused primarily on the design of the virtual world, the ability of participants to locate 
the Flolodeck@UF[ “classroom” area, and navigation in general. In addition, post-test interview questions probed the 
perceived overall usability experience. Analysis of the usability data suggests that participants' perceptions of the 
environments' ease-of-use are positive, with participants reporting their experiences as “pretty easy.” Flowever, a 
number of usability issues were identified. These usability issues are presented in Table 1 below. One identified 
usability issue stemmed from a lack of clear directions embedded in the virtual world to guide them to the areas they 
were asked to locate. However, both users were able to find the areas they were asked to locate in approximately the 
same amount of time and reported that finding the area was easy. In addition, participants were able to locate and 
activate the “screen” that presents the real-world classroom in the virtual world. They reported that this task was 
easy, that the view of the real-world classroom was clear and easy to see, and that zooming in on the screen and 
other areas of the virtual world classroom was easy. However, both participants had problems navigating their 
avatars up the stairs in the environment, with their avatars getting “stuck” and not being able to progress further. 
While our design provided for both ramps and stairs for navigating, participants seemed to naturally be drawn to the 
stairs. The more advanced participant quickly used a function to allow his avatar to fly in order to reach the intended 
destination, whereas the less advanced participant ultimately gave up and was unable to complete the task. In the 
post-test interview, both participants indicated a strong desire for more training and indicated that other users woidd 
likely experienced fewer challenges if given an appropriate amount of training. These findings suggest that while the 
Holodeck@UH is generally perceived positively and seems to have sufficient usability, more work is needed before 
we begin offering classes in mixed-reality. Findings from our preliminary analysis will help guide our next round of 
design considerations, and a more formal analysis of the data will guide our reflections as we return to our 
overarching design principles. 
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Table 1 


General usability’ findings, usability issues, and suggestions for revision 


General Usability 

Top Usability Issues Identified 

Suggestions for Revision 

• In general, participants rated the 
tasks they performed as “fairly 
easy” 

• Eye-tracking analysis provides 
powerful lens for inteipreting 
participant perceptions and behaviors 

• Using maximum variation sampling 
provided fast, meaningful results, 
even with n of 2 

• Insufficient visual cues in the 
environment for navigation 

• Some architectural elements invite 
actions that are not possible (i.e., 
stairs that cannot be climbed) 

• Lack of training diminishes learner 
experience 

• Develop and implement “signage” 
for environment that both orients and 
guides the user 

• With architectural elements, focus 
on function over form 

• The value of training cannot be 
understated -in-world training 
sessions as well as just-in-time and 
on-demand support are needed. 


Discussion 

Findings from our formative evaluation study, as well as from the general process of designing and 
implementing the Holodeck@UH has uncovered a number of useful lessons that we feel are of value to virtual 
educators considering using Open Source VWs platforms for their own immersive education initiatives. As a 
consortium of educational technologists who desire to relate to higher education the values of the Free/Libre 
Software movement (an initiative that has been criticized for its sometimes idealistic nature) it is important that we 
remain practical in our evaluation of the technologies we implement. In this section, we attempt to provide a 
measured discussion of what we learned in the process of implementing Free and Open Source VWs toolkits. While 
Open Source VW software like Open Wonderland and OpenSim may not cost money, this is not the reason we 
opted to use these toolkits. Instead, it was due to the freedom that Open Source software affords. Flowever, as 
readers will have learned from the history of our project (and to quote a former U.S. president), “Freedom isn’t 
free.” Implementing our system required skilled personnel with advanced expertise and skill. While our team is 
prepared for such challenges, we recognize that other VW educators may not have the necessary skills or time to 
invest projects such as this. 

While the findings from the research presented here are positive and encouraging, they are limited by a 
number of factors. First, the study sample may not be representative of the users who will ultimately be using the 
system. A majority of participants identified themselves as teachers. We must question if a sample of primarily 
students would have provided different responses. In addition, participants were not able to actually use the 
Flolodeck@UF[ or observe the system being used by actual students and teachers. Instead, they were presented with 
video vignettes of activities that are possible using the system. Some participants indicated in open-ended responses 
a desire to see how the system would function in a “real” instructional setting, and pointed out that without being to 
actually use the system, it was difficult to gauge the degree to which it is meeting its intended goal of enhancing 
social presence. Given the very early nature of our system, we view these limitations as challenges that should be 
approached in future iterations of our learning system and research design. 

The Flolodeck@UFI represents an evolutionary step for the mixed-reality solution originally developed by 
the MiRTLE team at the University of Essex. We have adopted their original design ideas, built on them in unique 
ways, and implemented our derived designs in a different VW software system. As we continue to advance our 
project, we systematically test the designs and the underlying system in order to gauge its viability and usefulness 
for providing an online learning platform that enhances social presence. Moving forward in our DBR trajectory, we 
will build lessons for teachers who have agreed to help us pilot test the Flolodeck with students enrolled in online 
courses, and to begin to implement learning analytics into the system to gauge how students are using it. Beyond 
that, we intend to field test the lessons we build in RW teaching and learning scenarios. The field tests will serve as 
opportunities to evaluate the Flolodeck as a distance learning platform, to collect and analyze data using our learning 
analytics solution, and to develop expertise orchestrating instruction and learning in our system. 
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The interaction opportunities presented by mixed-reality are fascinating and have ramifications for hybrid 
learning. Providing an online environment that allows for seamless collaboration and learning between participants 
in the real and virtual world is compelling. However, more work is needed to understand how to leverage the 
affordances of a mixed-reality learning modality to establish educational impact. As we continue our design-based 
research trajectory, we endeavor to pursue this goal. 
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Abstract 

Recently, tablet technologies have grown tremendously in popularity. They lend themselves to a myriad of 
learning modalities - visual, tactile, and auditory - and therefore may be well suited to use in schools and 
universities. While teachers in classes work to find useful applications for tablets in their teaching, students have 
already begun using them at home and, in secondary and higher education, in classes. Unfortunately, often when 
students use them for courses they frequently play with "apps" (applications that behave like software), rather than 
using the technology as a useful and powerful tool in the teaching and learning process. The current article 
addresses ways to use tablet technologies to their fullest potential as a important learning device in higher education. 

Introduction 

In the last decade, education has taken on a new rhythm in the lives of students and society. No longer is 
learning done at specific hours of the day in specific locations (generally schools and universities). Now education 
is something that occurs whenever a learner has a question or wonders about something and possesses a device to 
help him or her answer the query. Mobile learning is the present-day form in which education occurs. It is not only 
supported by mobile technologies, but also characterized by the mobility of the learners and the knowledge itself 
(Sharpies, Taylor, & Vavoula, 2010). Mobile learning devices include phones, laptop computers, and tablet 
technologies, such as the iPad and Android. Recent innovations in tablet technology have included computing 
devices that respond to touch, rather than, or in addition to, operating by mouse, stylus, track pad, etc. This form of 
operating is called gestural and the interfaces operate using Gestural Interface Technology (GIT). Gestural 
interfaces (Norman & Nielsen, 2010) include interactive whiteboards, smartphones, and most recently, tablet 
technologies. Only available since approximately 2010, gestural tablets have rapidly become popular with the 
public, with both adults and adolescents alike. These tablets have slowly made their way into schools and 
universities, with the institutions working once again to catch up with the technological innovation and find an 
educational use for it. Given the recency of tablets being used in the classroom, there is limited information as to the 
benefit of such practices on the teaching and learning process. There are however some features of tablets that 
appear to benefit students in a variety of ways. 

Due to their multitude of usability options, tablet technologies lend themselves to a myriad of learning 
modalities - visual, tactile, and auditory. The various makes, models, sizes and price ranges make the technology 
widely affordable and accessible for user needs. Unlike desktop computers or laptops, they adapt well to the needs 
of many different types of learners. The visual learner benefits from the wide range of applications on a tablet that 
has graphical user interfaces (GUIs) which represent ways of navigating a software, creating multimedia projects, 
and demonstrating understanding of concepts. Auditory learners find the sound options advantageous because text 
can be read, visual representations described, and of course, music and video are also included as features. 
Additionally, the most unique aspect of a tablet is the way the user operates it. Tablets are not operated via mouse 
or track pad, but rather contact is made right on the screen itself by the fingertips of the user. This feature is helpful 
for tactile learners who might have trouble remembering visual or auditory facts, processes, and instructions. By 
using their hands to proceed through websites, software and applications containing needed information, tactile 
learners possess a greater chance of being able to process and retain whatever it is that they need to learn. 

Generally, tablets are convenient devices that can handle many of the tasks of a small computer. Tablets 
and other portable devices have been marketed to everyone from young children to working professionals, and from 
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the stay-at-home parent/guardian to the world traveler. Accordingly, it's no surprise to see secondary school and 
college students bring tablets into their classes to use for coursework. Although having students use tablets in class 
to facilitate their learning sounds like a good idea, unfortunately learners often simply play with "apps" (applications 
that behave like software) rather than using the technology as a useful and powerful production tool in the teaching 
and learning process. Although a reasonably new topic in the literature, an assortment studies have been conducted 
in order to investigate the potential of tablet technologies in education. 

Literature Review 

In a longitudinal study conducted in Belgium, Courtois, DeGrove, Montrieux, Raes, De Marex, and 
Schellens (2013) found that secondary students used tablets in school because they perceived them as useful and 
enjoyable, rather than because their peers convinced them it was the newest technology to own. Sullivan (2013) 
reported that iPads supported both a student-directed approach to writing and an inclusive classroom. With early 
learners (preschool and early elementary students), Couse and Chen (2010) ascertained that children navigate and 
explore tablets with ease, feeling comfortable and learning by playing. Richardson (2012) contends that tablet 
technologies have increased the benefits of educational games in both the classroom and the home by making the 
games more accessible to P-12 students. Interestingly, Jones, Hall, and Hilton (2012) determined that children’s 
emotional responses to archived photographs were stronger when they viewed the pictures as physical items, while 
their cognitive responses were stronger when the photos were viewed digitally on tablets and/or flat screen 
televisions. In a related study, Vondracek (2011) reports that students found viewing video of lectures using tablets 
to be a useful resource. Using both tablet and stylus technologies, Kim, Kim, Choi, and Hahn (2013) recorded 
secondary students handwritten work during mathematics examinations in order to analyze it and provide timely and 
individualized feedback. In perhaps the most powerful study, Ferrer, Belvis, and Pamies (2011) found that use of 
tablets increased scores for low-performing students and decreased the academic effects of low socio-economic 
status in a public school in Spain. 

In addition to researching the use of tablet technologies in P-12 schools, several investigations were also 
conducted at the post-secondary level. Using an “active engagement metric,” Fagen and Kamin (2013) found student 
engagement in programming computer code to increase when using tablet technologies during college class 
discussions. Similarly, Nortcliffe and Middleton (2013) surveyed university students in the United Kingdom and their 
results support the idea that tablets foster autonomous, pervasive, student engagement. In their 2011 study, Loch, 
Galligan, Hobohm and McDonald utilized netbook tablets with volunteer students in college math courses. Their 
results cautioned that although the tablets promoted learner-centered pedagogy, the reduced cost of netbook tablets may 
not be worth the limited computing power that they provide. Conversely, in their 2012 research, Galligan, Hobohm and 
Loch concluded that students using tablets in a distance math course found that the technology facilitated assignment 
submission, efficient instructor feedback, and communication about math concepts between students and the instructor. 
Romney (2011) conducted a longitudinal study which revealed that undergraduate students who began their 
mathematics education using tablet technologies were more likely to continue their studies than were their non-tablet 
using peers. In 2012, Lavery found that college students using tablets for online creative writing and English classes 
participated more in online activities. She concludes that tablets can help to enhance face-to-face course experiences. 

In an important study, Milner-Bolotin and Antimirova (2010) discuss the need for instructor professional development, 
reflection, and practice with tablet technologies before using them in higher education physics classes. Finally, 
Manuguerra and Petocz (2011) advocate tablets as a way to change the teaching and learning process in higher 
education, specifically the way that students engage with each other and the content. 

Supporting the idea of tablets as transformative for schools and schooling, the NMC Horizon Report (2012) 
identifies mobile technologies, including tablet technologies as entering the mainstream of education at the present 
time. Given the prevalence of tablets in homes and in the workplace, and the rapidly growing appearance of this 
technology in classrooms, it’s clear that the predictions in the report have come to fruition. However, despite the 
presence of tablet technologies within education, their utility in the teaching and learning process remains a 
challenge. It is therefore worth discussing some applications useful for the classroom so that this innovation can 
become a more powerful tool for both instructors and students. 

Discussion 

Any current, useful, technology has a suite of productivity applications that include word processing, a 
database, a spreadsheet, and presentation software. Office Suite is one of these collections of applications and there 
is a free version as well as professional versions available for the Android tablet. Pages is the set of programs for 
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the iPad that is similar to Office Suite. These applications can allow the user to type word documents, create 
spreadsheets, and design presentations. Files in Pages are compatible with the Office Suite on a computer, laptop, 
and /or netbook. Given that productivity applications are the most commonly used programs on a computer, it 
would stand to reason that they would also be the most useful items on a tablet as well and make the technology 
more useful for both teacher and student productivity. 

Other types of tools available on tablets lend themselves to the organization of the learning process and 
students could benefit from gaining facility with them. Business Calendar helps keep track of schedules and 
appointments, while a program like Doodle allows members of a group to coordinate schedules for meetings and 
conferences. Financial/Budget application keeps track of spending practices and Time Tracker or Timesheet 
calculates the amount of time spent on a task or project. Also, programs like Evernote allow users to take notes 
while mind mapping applications such as MindMeister help coordinate and systematize brainstorming sessions. 
Tools such as these not only make productivity more efficient, but they also aid in project creation by helping 
students, and instructors, manage their time, assign tasks, and organize ideas, concepts, and processes. 

Additionally, multimedia applications are available for tablets for students who enjoy creative studies and 
hobbies. Musicians, audio engineers, or music aficionados would enjoy and use programs such as Pandora, Spotify, 
JetAudio, Wireless Mixer, Caustic, Equalizer and Music Volume. These “apps” allow users to produce, record, edit, 
mix, and clean up audio files. Multimedia and its associated softwares are particularly important because they give 
students another means of demonstrating “knowing” if they are not good at, or in addition to, the traditional reading, 
writing, and presenting to demonstrate understanding of a concept. Multimedia tools also give teachers the ability to 
offer alternative assignments and assessments so that students with a variety of who possess a variety of Multiple 
Intelligences (Gardner, 2008) are better able to engage in the learning process. 

While it is important to give students many avenues to understanding concepts and to demonstrate what 
they know, often these highly creative products produce huge files that necessitate a significant amount of storage 
space. Neither students not teachers consider virtual storage components for the technology products that they are 
assigning or designing. They would more likely think of storing their files on an external harddrive, or a flash drive 
(a risky move since flash drives truly are not meant for storage, but rather for transferring files from one place/user 
to another). A better alternative than either external harddrives or flash drives, is the concept of cloud technologies. 
Cloud technologies are virtual spaces hosted by a third party on a web service interface, outside of a individual’s 
personal computer, where information can uploaded and stored and/or transferred and programs, documents, 
resources etc. can be accessed and used (O’Brien & Marakas, 2011). Cloud technologies are becoming intimately 
linked to table technologies since many current tablets are being produced without the ability to connect any kind of 
external drive for transferring and/or saving files. Although cloud technologies are becoming increasingly common, 
students and instructors do not seem to avail themselves of this option as often as they could or should. It would 
serve both students and educators well to avail themselves of cloud storage technologies more. One of the most 
popular types of cloud storage is Dropbox.com. It is a virtual storage space and offers 2GB of online storage. All a 
user needs to do is sign up in order to open a free account. For additional storage capacity, in the 5GB range, there 
are other options, such as Google Drive, Ubuntu One, OpenDrive, and/or Cloud Drive. While SkyDrive from 
Microsoft offers 7GB, for users who need even more storage, there is 4Shared.com - which provides 15GB of 
online storage. If a user has multiple online storage accounts and does not want to log into every account to see 
where things are located, then an interface, such as ZeroPC [www.zeropc.com], can allow him/her to login to all 
his/her accounts at once. This option also gains the user approximately 40-45GB of online storage, adding to the 
overall storage capacity of a tablet. These are all very helpful options since they are free and would allow students 
to share files between devices & computers, thus increasing their ability to work more easily with others in groups 
or teams, regardless of whether they meet face-to-face or not. (For virtual meetings, programs such as Skype,ooVoo 
and Facetime work wonderfully via tablet technologies.) Technical usability issues are key when teachers and 
students enter the digital realm since learning and creating projects can be frustrating by themselves. Adding a layer 
of technical frustration on top of the normal challenges that occur in the learning process can often shut down a 
student (or educator) completely and turn them off to either the concepts being learned or the medium through 
which they are learning, or both. Using easily accessible common digital spaces through which students can share 
work is important when integrating technology into education. 

Perhaps the most cutting edged of all the tablet applications is the ability to communicate between types of 
hardware. Some apps, such as Dolceri, offer communication between iPad tablets and laptops via Bluetooth. With 
the laptop connected to a projector, a user could then have a somewhat interactive screen with a tablet serving as the 
“remote”. A teacher could simply walk around the classroom and give a presentation or mock lesson with the 
added ability to work on the iPad and have the results displayed on the screen at the front of the room. Options for 
more interactive types of learning could easily occur using tablet technologies as students share whatever is on their 
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tablet with the rest of the room via a projection application. This type of feature is particularly powerful with peer- 
editing or group review and analysis of student-produced work. Allowing everyone in the class to see the same 
piece of work and interactively discuss and work on it is an extremely engaging and useful pedagogical technique. 

Regardless of what the latest technological device may be, the key for enhancing the teaching and learning 
process is to use the device and its associated programs to their fullest extent. Given the newness of gestural tablet 
technologies, the potential of these tools are in the process of being discovered. There remain many facets of using 
tablet technologies in education yet to explore. 


Future Directions 

Tablet technologies may well usher in a new way of teaching and learning. Less expensive than desktops 
and laptops, tablets are also more portable and provide easier access to applications such as email and Internet 
searching. Conversely, they are less user-friendly when working on large projects, papers, or multimedia products. 
Smartphones are also not useful for completing large-scale work, but they are more portable than tablets. However 
tablets are infinitely less expensive because they do not require a monthly fee to remain in use. Given their 
limitations, and clear advantages, tablet technologies may well be an innovation that becomes pervasive in schools 
and schooling. 

In his writing about technological innovations and education, Thimvathukal (2013) states that, “If we don't 
change, others will change us and render us obsolete or, worse, irrelevant.” It will do harm to students, teachers, and 
the educational process in general if pedagogy does not keep pace with technology. This challenge is even more 
urgent than in previous eras due to the nature and preferences of the students who are currently in schools. Gardner 
and Davis (2013) have labeled the current cohort of youth the “App Generation.” They claim that these individuals 
are immersed in all facets of digital media, which affect their sense of identity, intimacy, and imagination. These 
attributions are quite compelling and educators today must be cognizant of who they are teaching. 

While challenging, this set of circumstances is alive with potential. Jones and Thimvathukal (2012) call 
these situations “possibility spaces.” They consider possibility spaces to be everywhere, in everything, with new 
ideas, innovations, and techniques just waiting to be discovered. Given this framework, several aspects of using 
tablets in the classroom have yet to be investigated. These areas include use of tablets on a large-scale basis in a 
school or school district, longitudinal studies of using tablets in the classroom, the effectiveness of using tablets in 
teacher preparation program field placements, and most importantly, measuring the learning gains (if any) from 
utilizing tablet technologies in the teaching and learning process. Studies such as these should and will likely be 
undertaken in the next several years and our knowledge of and facility with using tablets in the classroom will likely 
both broaden and deepen. Only time will tell if tablet technologies in the classroom are here to stay. 

References 

Courtois, C., DeGrove, F., Montrieux, H., Raes, A., De Marex, L., & Schellens, T. (2013). Push or pull? A 

longitudinal survey study into the acceptance of tablets in secondary education. Proceedings of the 7 th 
annual International Technology’, Education and Development Conference, 356-364. 

Couse, L.J., & Chen, D.W. (2010). A tablet computer for young children? Exploring its viability for early childhood 
education. Journal of Research on Technology’ in Education, 43 (7), 75-98. 

Galligan, L., Hobohm, C., & Loch, B. (2012). Tablet technology to facilitate improved interaction and 

communication with students studying mathematics at a distance. Journal of Computers in Mathematics 
and Science Teaching, 31 (4), 363-385. 

Gardner, H. (2008). Multiple Intelligences: New horizons in theory’ and practice. New York, NY : Basic Books. 
Gardner, H. & Davis, K. The App Generation: How today's youth navigate identity, intimacy, and imagination in a 
digital world. New Haven, CT: Yale University Press. 

Fagen, W. & Kamin, S. (2013). Measuring increased engagement using tablet PCs in a code review class. 

Proceedings of the 44 th annual Association for Computing Machinery’ Special Interest Group on Computer 
Science Education, 465-470. 

Ferrer, F., Belvis, E. & Pamies, J. (2011). Tablet PCs, academic results and educational inequalities. Computers & 
Education, 56(7), 280-288. 


592 



Jones, S. Hall, L., & Hilton, J. (2012). Exploring children's 'indexical encounter' with real and digitized archive 
photographs using tablet and large flat screen technologies. Proceedings of the 11th International 
Conference on Interaction Design and Children, 30-39. 

Jones, S.E. & Thiruvathukal, G.K. (2012). Codename revolution: The Nintendo Wii platform. Cambridge, MA: 

MIT Press. 

Lavery, H. (2012). Breaking down barriers with tablet technology for teaching. Educause Review Online. Retrieved 
May 30, 2013 at http://oro.open.ac.uk/34230 . 

Loch, B., Galligan, L., Hobohm, C., & McDonald, C. (2011). Learner-centred mathematics and statistics education 
using netbook tablet PCs. International Journal of Mathematical Education in Science and Technology, 
42(7), 939-949. 

Kim, Y., Kim, C., Choi, H., & Hahn, M. (2013). Modeling student’s handwritten examination data and its 

application using a tablet computer. Lecture Notes in Electrical Engineering IT Convergence and Security, 
215,861-865. 

Manuguerra, M. & Petocz, P. (2011). Promoting student engagement by integrating new 

technology into tertiary education: The role of the iPad. Asian Social Science, 7(77), 61-65. 

Milner-Bolotin, M. & Antimirova, T. (2010). Hp grants for tablet technologies in science teaching: From dream to 
innovation. Proceedings of the 4th International Technology, Education and Development Conference, 
3432-3443. 

NMC (20120. Horizon report: 2012 higher education edition. Retrieved June 1, 2013 at 
http://www.nmc.org/pdf/2012-horizon-report-HE.pdf . 

Norman, D.A. & Nielsen, J. (2010). Gestural interfaces: a step backward in usability. Interactions, 17(5), 46-49. 

Nortcliffe, A. & Middleton, A. (2013). The innovative use of personal smart devices by students to support their 
learning. In Wankel, L.A. & Blessinger, P. (Eds.), Increasing student engagement and retention using 
mobile applications: Smartphones, Skype and texting technologies (Cutting-edge technologies in higher 
education Volume 6). Bingley, UK: Emerald Group Publishing Limited. 

O’Brien, J.A. & Marakas, G.M. (2011). Computer software, management information systems (10th ed.). New 
York, NY: McGraw-Hill. 

Richardson, W. (2012). Gaming gains respect. District Administration, 48(6), 44-47, 50-51. 

Romney, C.A. (2011). Tablet PC use in freshman mathematics classes promotes STEM retention. Proceedings of 
the 41 s ' annual Frontiers in Education Conference, F1J-1 - F1J-7. 

Sharpies, M., Taylor, J., & Vavoula, G. (2010). A theory of learning for the mobile age. In Bachmair, B. (Ed.), 
Medienbildung in neuen Kulturrdumen. Wiesbaden, Germany: VS Verlag fur Sozialwissenschaften. 

Sullivan, R. (2013). The tablet inscribed: Inclusive writing instruction with the iPad. College Teaching, 61(7), 1-2. 

Thiruvathukal, G.K. (2013). Who needs tablets? We do. Computing in Science & Engineering, 15(7), 4-6. 

Vondracek, M. (2011). Screencasts for physics students. The Physics Teacher, 49(2), 84. 


593 


Assessing Online Learning Readiness 


Jian Su Searle and Michael L. Waugh 
University of Tennessee at Knoxville 
Knoxville, TN 37996. 

E-mail: jsul@utk.edu 

Keywords: distance education; online learning readiness 


Abstract 

This paper reviews the current status of research on the online learning readiness of students, and presents a 
rationale and proposal for the development of a new survey instalment to establish an online learning readiness 
profile for all potential online learners. These profile data would help enable instructors and researchers to establish 
how student learning readiness profiles might relate to student success or failure in online courses and programs, 
thus contributing to improved online courses and program design. 

Introduction 

This paper reviews the current status of research on the online learning readiness of students. Based on the 
review, the paper presents a rationale and proposal for the development of a new survey instalment to establish a 
pre- participation readiness profile for potential and/or new online learners. An online student readiness instalment 
that is based upon critical readiness factors identified by earlier researchers should be developed to gather data from 
online program applicants, in order to determine whether or not a student readiness profile can predict student 
success in online programs. 

The data from such a readiness instalment could be utilized in a variety of ways to assist online program 
developers in providing the best possible online learning experiences for their students. Such profile data could be 
matched with student outcome data in order to determine their predictive validity. Moreover, these data should be 
reported widely to help establish the validity of the critical factors proposed by previous online learning researchers 
and those interested in student attrition. If one or more clear relationships can be documented between specific 
critical factors and student outcomes, then such results could be examined in a controlled experimental context to 
establish the possibility of their causal influence on student success. 

Literature Review 

With the rapid growth and popularity of the Internet, distance education, specifically online education, has 
also grown rapidly, as institutions of higher learning have sought to keep pace with this new technological capability 
and the rapid growth in acceptance of these technologies by working adult students. Large numbers of higher 
education institutions are now offering online courses or programs to students, and this growth appears to be 
continuing at a rapid pace (Allen & Seaman, 2013). Such online academic offerings bring many benefits to learners 
(Carr, 2000; Mayes, Luebeck, Ku, Akarasriworn, & Korkmaz, 2011) such as flexibility, convenience, affordability, 
and applicability. However, along with this rapid growth, numerous challenges also accompany the emergence of 
successful online courses and programs. 

Student Persistence and Attrition in Online Programs 

Several researchers have identified what they describe as a series of crucial factors in students' decisions to 
persist in academic programs (Bean & Metzner, 1985; Tinto, 1975, 1987, 1993). This attrition research in general 
education started before the emergence and popularity of online learning. A number of subsequent online learning 
studies (Carr, 2000; Conceipao & Lehman, 2013; Rovai, 2003) examined the older student attrition work and 
concluded that beyond those variables that are critical for all students to be successful in academic programs, other 
factors now exist that particularly affect the success of online student. Together these studies provide a theoretical 
framework that can help researchers understand why online students might or might not be successful in online 
academic programs. 
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Online Learning Readiness Surveys 

With the rapid growth of online learning as well as the increased attention paid to attrition rates in online 
programs over the past decade, a number of researchers (Elliott, Hall, & Meng, 2008; Davis, 2010; Miller, 2005; 
Plata, 2013; Schmidt, Khiewnavasongsa, & Newton, 2010; Shraim & Khlaif, 2010; Smith, Murphy, & Mahoney, 
2003; So, 2008; Warner, Christie, & Choy, 1998) have stressed the importance of understanding students' readiness 
for learning online, for the purpose of potentially preventing high online attrition rates. Different learners starting an 
online program together very possibly have different previous online learning experiences. An online learner’s 
initial readiness level to learn using a variety of telecommunication technologies is clearly an important element in 
his/her potential success as an online student. Based on this understanding, in recent years, more and more online 
programs such as Northern Illinois University Online and Florida Gulf Coast University (see Table 1) have provided 
their prospective students with online readiness instruments to help them self-assess their online learning skills and 
abilities. This paper identifies and describes these existing measuring instruments utilized for assessing online 
learners' readiness. 


Institutions 

The Columbia College 

The Community College of 
Baltimore County 
Florida Gulf Coast 
University 

Foothill College 

Northwest Florida State 
College 

Northern Illinois University 
Online 

Our Lady of the Lake 
College 

The Pennsylvania State 
University 

St. Louis Community 
College 

University of Houston 
Distance Education 
The University of North 
Carolina at Chapel Hill 


Table 1 

Samples of Online Readiness Survey Instruments 

Instruments 

Online Campus Self-Assessment Quiz 

http://www.ccis.edu/online/admissions/selfassessment.asp 

Distance Learning Self-Assessment Test 

https://www.ccbcmd.edu/distance/assess.html 

Technology Skills Self-Assessment Survey 

Distance Learning Self-Readiness Assessment 

http://www.fgcu.edu/support/techskills.html 

Online Student Readiness Assessment 

http://www.foothill.edu/fga/pre assessment. php 

Distance Learning Self-Assessment 

http://ecampus.nwfsc.edu/selftest.CFM 

Online Student Readiness Survey 

http://www.niu.edu/niuonline/ 

Are You Ready for Distance Learning? 
http://www.ololcollege.edu/DL Readiness Survev.htm 
Student Self-Assessment for Online Learning Readiness 

https://esurvev.tlt.psu.edu/Survey.aspx7sA246aa3a5c4b64bb386543eab834f8e75 
READI Assessment (Readiness for Education At a Distance Indicator) 
http://stlcc.readi.info/ 

Test of Online Learning Skills 
http://distance.uh.edu/online learning.html 
Online Learning Readiness Student Self-Assessment 

http://www.unc.edu/tlim/ser/ 


A large percentage of the existing e-learning readiness measurement models were developed for 
organizations in non-academic settings, mainly for business organizations. These models focus on main criteria such 
as facilities/infrastructure for e-learning, management, organization of e-learning function/department, learners' 
characteristics, and the e-learning process (Al-Osaimi, Alheraish, & Bakry, 2008; Chapnick, 2000; Fetaji & Fetaji, 
2009; Goi & Ng, 2009; Haney, 2002; Ho, 2009; Lopes, 2007; Schreurs & Al-Huneidi, 2012; Slick, 2001). One of 
the most cited models is the Chapnick (2000) needs assessment model for assessing organizational readiness for e- 
learning. Chapnick defined the e-learning readiness measurement as a process for determining the gap between what 
students know and what they need to know. She listed 66 survey questions and grouped them into eight categories: 
psychological, sociological, environmental, human resources, financial, technology skill (aptitude), equipment, and 
content readiness. The purpose was to enable different stakeholder groups to conduct the readiness measurement by 
examining a different combination of items. Her model helps stakeholders assess how ready their companies are by 
identifying their successful areas and the areas that need improvement. 

The above organizational e-learning readiness models and related instruments are not entirely suitable for 
application into educational settings. The online readiness assessment tools that have thus far been developed for 
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academic institutions appear to be relatively primitive compared to the e-readiness instruments for business 
organizations. The concept of readiness for online learning in academic settings was proposed by Warner, Christie, 
and Choy (1998) in a study on the readiness of students within the Australian vocational education and training 
sector for participation in online learning environments. The authors defined readiness for online learning in terms 
of three aspects: (a) students' preferences for the form of delivery as opposed to face-to-face classroom instruction, 
or the provision of print-based pre-packaged resource materials, (b) students' confidence in using electronic 
communication technologies for learning and, in particular, competence and confidence in the use of Internet and 
computer-mediated communication, and (c) students' ability to engage in autonomous learning. The students who 
participated in Warner et al.'s study demonstrated a low preference for online learning, low confidence, little 
experience with online learning, and poor skills for self-directed, autonomous learning. 

Mattice and Dixon (1999) developed a survey to assess students' interest and readiness in online learning. 
Their survey consisted of student readiness, student technology, and student interest subscales. Their student 
readiness measurement included students' self-direction, orientation to time, preferences for feedback, and students' 
previous experience with distance education. The technology subscale measured students' access to technology such 
as the Internet and email. The interest subscale asked about students' future online enrollment planning. 

McVay Lynch (2000, 2001a, 2003) developed a 13-item instrument for measuring readiness for online 
learning. The instrument identified two factors, students behavior and attitudes, as the potential predictors to online 
learner readiness. Her survey items were rated by respondents on a 4-point Likert scale (see Table 2). The McVay 
instrument was reliable and applicable to research and practice associated with learner readiness in an online 
learning setting. The McVay instrument has been considered as an important and useful online readiness assessment 
tool and has exerted a great influence on subsequent readiness studies. 

Table 2 

McVay Lynch's (2000) Online Readiness Survey Items 

1. 1 am able to easily access the Internet as needed for my studies. 

2. 1 am comfortable communicating electronically. 

3. 1 am willing to actively communicate with my classmates and instructors electronically. 

4. 1 am willing to dedicated 8 to 10 hours per week for my studies. 

5. 1 feel that online learning is of at least equal quality to traditional classroom learning. 

6. 1 feel that my background and experience will be beneficial to my studies. 

7. 1 am comfortable with written communication. 

8. When it comes to learning and studying, I am a self-directed person. 

9. 1 believe looking back on what I have learned in a course will help me to remember it better. 

10. In my studies, I am self-disciplined and find it easy to set aside reading and homework time. 

11.1 am able to manage my study time effectively and easily complete assignments on time. 

12. As a student, I enjoy working independently. 

13. In my studies, I set goals and have a high degree of initiative. 


Smith, Murphy, and Mahoney (2003) conducted a survey study to examine the reliability and validity of 
the McVay instalment. Their study on the instalment resulted in a two-factor staicture: comfort with e-learning and 
self-management of learning. The study concluded that the McVay survey instrument of readiness for online 
learning serves the purpose of assessing students' online learning dispositions and preferences, especially on the 
above two factors. However, they also stressed the need for further research in order to establish the predictive 
validity of the McVay instalment. 

Bernard, Brauer, Abrami, and Surkes (2004) developed a new survey based on the McVay instrument and 
expanded the original 13 items to 38 items for the purpose of predicting online learning achievement of students. 
Their survey examined students' self-efficacy with technology and students' ability to online learning success. This 
study again recognized the strengths of the McVay instalment. Also, like several previous studies, the limitations of 
the McVay instalment were discussed by the authors. 

Kerr, Rynearson, and Kerr (2006) developed their own online readiness instalment, based on several 
existing instalments. The Kerr et al. instalment highlighted learners' self-esteem, learning styles, metacognitive 
reading strategies, academic intrinsic motivation, and academic locus of control. Their survey constiucts included 
students' need for online learning, computer skills, academic skills, and independent and dependent learning. 

Dray, Lowenthal, Miszkiewicz, Ruiz-Primo, and Marczynski (2011) incorporated the aspects of the digital 
divide and developed a survey based upon several existing surveys. Their 32-item survey consisted of two scales: 
learner characteristics and technology capabilities. Moreover, the authors suggested focus groups and interviews as a 
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necessary step in the instrument design process. They believed that the validity of their instrument could be 
increased by collecting prospective online learners' interpretations of the survey items. 

Hung, Chou, Chen, and Own's (2010) developed a new instrument and conducted a study on learner 
readiness of college students in Taiwan. Their 18-item instrument identified five constructs of online readiness: self- 
directed learning, motivation for learning, learner control, computer/Internet self-efficacy, and online 
communication self-efficacy. In their study, students reported high levels of readiness in the factors of 
computer/Internet self-efficacy, motivation for learning, and online communication self-efficacy, while low levels of 
readiness in learner control and self-directed learning. Their study also found that, compared to lower grade 
students, higher grade students showed significantly higher readiness in self-directed learning, online 
communication self-efficacy, motivation for learning, and learner control. 

Later, Kaymak and Horzum (2013) incorporated Hung et al.'s (2010) survey instrument in order to explore 
the relationship between students' readiness levels for online learning and their perceived structure and interaction in 
online learning environments. These authors concluded that students' online learning readiness was positively 
related to their online interactions but negatively related with perceived structure. The authors pointed out that 
online learning readiness was important regarding the structure that affects students' interaction as well as learning 
outcomes. 


Table 3 

Summary’ of Studies Using Readiness Survey Instruments 


Study 


Constructs 


Akaslan and Law(201 1) 
Bernard, Brauer, Abrami, and 
Surkes (2004) 

Dray, Lowenthal, Miszkiewicz, 
Ruiz-Primo, and Marczynski 
( 2011 ) 

Hung, Chou, Chen, and Own 
( 2010 ) 

Kerr, Rynearson, and Kerr 
(2006) 

Mattice and Dixon (1999) 
McVay (2000, 2001a, 2003) 


Smith, Murphy, and Mahoney 
(2003) 

Smith (2005) 

So (2008) 

Warner, Christie, and Choy 
(1998) 

Watkins, Leigh, and Triner 
(2004) 


Technology, people, content, institution, acceptance, training 

Beliefs about distance education, confidence in prerequisite skills, self-direction 

and initiative, and desire for interaction 

Learner characteristics, and technology capabilities 


Self-directed learning, motivation for learning, learner control, computer/Internet 
self-efficacy, and online communication self-efficacy 

Computer skills, independent learning, need for online learning, academic skills, 
and dependent learning 

Student readiness, student assess to/use of technology, and student interest in 
distance education 

Students' comfort with computer skills and components of online learning, and 

students' independence as learners 

Usefulness, self-efficacy, willingness, challenges 

Comfort with e-learning, self-management of learning 

Student dispositions and preferences in online learning 
Government and public support, technical competency, e-learning materials, 
training opportunity, attitude towards e-learning, competency of peers 
Students' preferences for the course delivery format, student competence and 
confidence in using computer-mediated communication for learning, and student 
ability to engage in autonomous learning 

Technology access, online relationships, motivation, online video/audio, internet 
discussions, importance to success 


The existing online student readiness instruments (Akaslan & Law, 2011; Dray et al., 2011; McVay Lynch, 
2000, 2001a, 2003; Watkins, Leigh, & Triner, 2004) usually contain multidimensional measurement scales that 
mainly focus on the following variables: (a) students' general learner characteristics, such as perception of 
interpersonal communication skills, (b) basic technology skills, such as ability to use specific applications in specific 
ways, and (c) their self-management of learning, or self-directed learning. Table 3 presents a summary showing the 
constructs measured by each instrument. 

A range of online students’ general learner characteristics, such as motivation, attitude, confidence, 
independence, and communication, have been recognized as significant factors that determine online learner 
participation, interaction, and satisfaction (Brahmasrene & Lee, 2012; Hung et ah, 2010; McVay Lynch, 2001a). For 
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example, student perceived interpersonal communication skills, or social ability, directly influences online learning 
readiness. Brahmasrene and Lee (2012) examined three determinants of intent to continue using online learning: 
perceived social ability, online learning readiness, and perceived usefulness. Their study found that students’ 
perceived social ability exerts an important influence on online learning readiness, and that in turn, students’ online 
learning readiness shows a positive effect on their intent to continue using online learning. One example of such 
items from the McVay instrument is: "lam comfortable communicating electronically." 

Students' basic technology’ skills/ capabilities, or comfort with technology ;, is a key factor which is included 
in all the existing online readiness literature. There is no denying the importance of computer skills in online 
learning. As a matter of fact, this technology readiness factor is important to students in all educational settings. 
Parasuraman (2000) presented the Technology Readiness Index that investigates an individual’s preferences and 
beliefs about adopting and using innovative technology, in order to measure his/her level of techno-readiness. 
Individuals who are ready to use technology appear to be more likely to try it (Parasuraman, 2000). Research has 
also shown that students’ technology readiness levels when they enter college are significantly associated with their 
attrition levels (Elliott, Hall, & Meng, 2008; Ratliff, 2013). McVay Lynch (2000) described the technology factor as 
a readiness for engagement with the particular form of resource-based flexible learning delivery in an online setting. 
Similar arguments are also found in general learning literature (Sadler-Smith & Riding, 1999) exploring an 
association between cognitive style and learner comfort with different forms of resource-based learning materials. 
One example of such items from the McVay instrument is: " lam able to easily access the Internet as needed for my 
studies." 

Another generally recognized key factor in online learning readiness is students' self-management of 
learning, or self-directed learning. This factor originated from Knowles' (1975) concept of self-directed learning, 
defined, in a broad way, as a process “in which individuals take the initiative ... in diagnosing their learning needs, 
formulating learning goals, identifying human and material resources for learning, choosing and implementing 
appropriate learning strategies, and evaluating learning outcomes” (p. 18). Knowles' concept was further developed 
into the self-directed learning readiness scale by Guglielmino (1977) for the assessment of students' learning needs. 
After that, Garrison (1997) presented a more comprehensive model on self-directed learning that aimed to promote 
student learning outcomes. Because of its particularly flexible features, the online learning environments have been 
considered suitable environments for learners to take charge of their own learning (Carr, 2000). In the past decades, 
online learning researchers have identified and recognized self-directed learning as a critical element in online 
learning. Online learners who are more self-directed in making decisions regarding their learning tend to be more 
successful in online learning in higher education, where a majority of the learner population is adult learner, 
(Anderson, 1993; Calder, 2000; Lin & Hsieh, 2001; Pachnowski & Jurczyk, 2000). Many online educators (Evans, 
2000; Warner et ah, 1998) have recognized self-direction as a necessary prerequisite for effective resource-based 
learning in distance education and flexible delivery. Several studies (Hung et ah, 2010; McVay Lynch, 2000) 
included a set of self-directed learning items in their online learner readiness survey instrument. One example of 
such items from the McVay instrument is: "In my studies, I set goals and have a high degree of initiative . " 

Limitations of Current Instruments 

The instillments developed over the last decade have focused on measuring different aspects of online 
readiness. Some focused on students' general learner characteristics and technical experience (McVay Lynch, 2003; 
Watkins et ah, 2004); while a few others focused on students' individual learning styles (Hung et ah, 2010; Miller, 
2005). None of the previous readiness survey instruments have proven to be as comprehensive, rigorous, and 
effective as initially claimed by the instrument developers. For example, multiple, related studies (Bernard et ah, 
2004; Dray et ah, 2011; Smith, 2005) have shown that McVay’s instrument does not include many factors that 
appear to be significantly related to online learner readiness. 

Although some researchers claim that their instruments are generalizable across contexts, each instrument 
typically has been designed by researchers for individualized research settings. This has resulted in potential 
limitations in the depth and breadth for most of the existing instruments. Previous studies seem to indicate an overall 
recognition of the limitations of the existing readiness surveys. They also acknowledge the need to expand the items 
to enhance reliability and validity. 

Most previous studies of online learner readiness have included few or no student demographic variables. 
However, student demographic variables, such as gender, age, and employment status, seem to play important roles 
related to student success in online learning. Demographic variables could even be the essential predictors, 
especially for nontraditional students who constitute a large percentage of the higher education online learner 
population. 
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Readiness research in the online learning area has been relatively recent with very few longitudinal 
analyses. As a result, the overall design of readiness surveys has been relatively fragmented and inconclusive. The 
current research on how to determine the key factors related to online learners' readiness and how these factors 
affect online learning are thus inconclusive, possibly due to the dynamic and complex nature of online learning. 

Also, with the emergence of new technology tools over time, some of the specific content of the existing readiness 
instalments is no longer relevant. As a result, many of the existing readiness instruments now have a somewhat 
limited practical application. 

Need for a New Instrument 

Because these earlier readiness instalments were developed in specific contexts and may be in need of 
significant revision, a new student readiness instalment should be developed that would be appropriate across a 
broader range of instalment contexts. Such an instalment would enable a series of studies to establish how a student 
readiness profile might relate to student success or failure in an online program. Even though each individual study 
might be a single case event, a series of case reports would help to validate the critical student success variables, and 
might contribute to a better understanding of how programs can enhance student success by utilizing such 
information. 

With the ongoing popularity of online education, there will be a growing need for both online educators 
and learners to keep developing more comprehensive and effective measures of learner readiness. A more 
comprehensive approach would better serve the purpose of assessing and re-assessing learner readiness. Further 
efforts are especially needed to establish predictive validity of the readiness instalments. The ultimate goal should 
be to develop standard and universally agreed-upon readiness assessment tools, and to establish graphical, 
conceptual online learning readiness models, especially for the needs of higher education settings. 

Providing Feedback and Follow-Up to Survey Respondents 

Some of the existing instalments provide instant feedback to students, with or without scoring. Typically 
students receive feedback immediately after their submission of the survey. Northern Illinois University and the 
Community College of Baltimore County provide examples of this approach. Students are asked to provide 
information regarding critical variables, and then, based upon their data, results are offered to students as an 
estimation of the degree to which online learning might be a good fit to their circumstances, learning style and 
lifestyle. Very few instalments do not require a submission, but provide scoring guidelines to students instead. One 
example is the scoring guide provided by Our Lady of the Lake College: 3 points for every option “a” selection, 2 
points for every b selection, and 1 point for every c selection. In most cases, the survey results include a total score 
and some recommendation, as feedback statements. The students are immediately given rather generic advice about 
the need to strengthen skills in some areas, or alter factors in their lives to ensure that they can better engage 
themselves in the academic programs. This is immediate feedback for the students based upon the presumption that 
the factors identified by the earlier researchers are relevant and critical to student success in online learning, in hopes 
that the students will utilize such information to address these issues in their personal lives in such a manner as to 
make them more successful online students. 

A different approach used by other researchers is to have potential or new online students provide 
information about how their personal situation is characterized in terms of these critical variables, so that the 
academic program organizers can use the data to adapt or enhance the program to better suit the collective needs of 
their students. For example, based on the survey results, an orientation program can be designed to especially suit 
one particular group of students. In this way, the orientation can best prepare the students for online learning, which 
eventually helps hopefully reduce their future attrition from the program. 

Some institutions offering online education have started to arrange orientation training to help students to 
better prepare and therefore improve the probability of their success as online learners. Currently theses orientation 
programs are delivered in various forms: formal or informal, mandatory or optional, face-to-face or online. The 
common topics covered in these orientations include guidance in online etiquette, technological skills, available 
guidelines and resources, and re-assessment of students' readiness for online learning. McVay Lynch (2001b, 2003) 
designed an online orientation program to offer training on independent, self-directed learning in an online learning 
environment. The orientation was highly recommended for students who were new to online learning. A few 
universities require all new online students to take their face-to-face orientation. For example, an online master's 
degree program offered by the College of Allied Health Sciences of the East Carolina University requires a 
mandatory 2-to-3-day orientation attendance for all new online students during the designated on-campus orientation 
sessions. The goal is to increase students' technological competence and confidence in order to increase their online 
readiness so they will have a higher probability of becoming successful online learners. Researchers also 
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recommend that institutions design and offer a faculty development/orientation program for faculty new to online 
teaching, and argue that this is equally important to online success (McVay Lynch, 2003; Miller, 2005). 

Conclusion 

In summary, a new online student readiness instrument should be developed based upon the attributes 
found in common on the previous instruments and incorporating additional relevant demographic characteristics and 
excluding context-specific or temporal-specific, that is, evolving technologies, characteristics. Using such an 
instrument will make it easier for program developers to look at how these pre -program student and program design 
characteristics correlate with program success. Most importantly, such an assessment should help enable program 
designers/developers to improve their instructional design in order to ultimately increase online program 
effectiveness. 
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Introduction 

The Joint Commission on Health Education and Promotion Terminology (2002) defines health education as 
“any combination of planned learning experiences based on sound theories that provide individuals, groups, and 
communities the opportunity to acquire information and the skills needed to make quality health decisions” (p. 6). 

To that effect, health professionals create health education interventions to provide learners with the knowledge and 
skills that enable them to make decisions conducive to reaching and maintaining optimal health (Glanz, Rimer, & 
Viswanath, 2008; Simons-Morton, Greene, & Gottlieb, 1995). 

The channels of delivery of the health education interventions varies to include face-to-face, print, mass 
media, and to technology-based communication (Glanz et al., 2008). However, health communication and 
information delivered through the use of technology is a growing trend in health known as ehealth (Pagliari et al., 
2005). Moreover, the popularity and the ubiquitous nature of mobile devices (Chen, Chang, & Wang, 2008) have 
opened the way to the delivery of health information through mobile devices known as mhealth. The World Health 
Organization (2011) reports that the use of both ehealth and mhealth is on the increase because it facilitates the 
dissemination of the interventions on a wide scale, and yet it enables them to be tailored to the specific needs of the 
learners with methods as complex as websites or as simple as emails (Norman et al., 2007). In fact, Glanz et al. 
(2008) describe the use of technology in health education as “an important part of the armamentarium of strategies 
for health education and health behavior” (p. 8). 

eHealth and mHealth education interventions offer several advantages to the learner, especially when they 
incorporate interactivity, collaboration, and real-time support (Fukuoka, Kamitani, Bonnet, & Lindgren, 2011; 

Kamel Boulos & Wheeler, 2007). This active involvement makes learning meaningful and motivates learners to 
initiate health behavior change (Banas, 2009). However, few of the health education studies reported in the 
literature were able to prove their effectiveness due to poor development and implementation (Kreps & Neuhauser, 
2010). Moreover, research has not proved yet which, if, and for whom ehealth interventions are effective (Baker et 
al., 2010). Nevertheless, investing in ehealth and mhealth education interventions is worthwhile if these 
interventions can show effectiveness in reaching the desired outcomes. 

In order to be effective, ehealth and mhealth education interventions need to follow certain guidelines that 
stem from the disciplines that guide them. In that respect, health professionals draw from a variety of disciplines, 
specifically the discipline of behavioral sciences and the discipline of education. The discipline of behavioral 
sciences informs health professionals about why people behave the way they do in health and offers guidance for 
behavioral change (Bandura, 2004; Glanz et al., 2008; Prochaska & DiClemente, 1986; Rosenstock, Strecher, & 
Becker, 1988). On the other hand, the discipline of education informs health professionals about how people learn 
(Driscoll, 2005; Gredler, 2001; Reigeluth, 1999). 
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From the education perspective, the health education literature is not explicit on how learning principles fit 
in the picture of the creation of health education interventions. More so, the health behavior theories or models that 
generally guide health education interventions, describe behavioral change without much emphasis on the learning 
process that result in the behavioral change (Begoray & Banister, 2005; Timmreck et al., 2010). 

However, developing education materials that lead to behavioral change, health or otherwise, can benefit 
from guiding principles that improve their effectiveness and appeal to learners. These guiding principles, which 
stem from learning theories and instructional models, are portrayed through instructional design models that explain 
the phases of instruction design and provide rigor to produce effective, efficient, and relevant instruction (Gustafson 
& Branch, 2002; Gustafson & Branch, 2007). In addition, these instructional design models serve as “conceptual 
and communication tools to visualize, direct, and manage processes for creating high quality instruction” (Gustafson 
& Branch, 2002, p.l). 

A review of the ehealth and mhealth education literature reveals that few studies report on the instructional 
design process used in the interventions. For example, Stevens et al. (2008) described the design, development, and 
implementation of a web-based intervention addressing weight loss. A team of experts selected a theoretical 
framework, determined the scope and the objectives of the website, designed the interface and translated the content 
into interactive modules. A paper prototype was presented to the team for approval and sign off. After development, 
the intervention was pilot tested and data on usability was collected. Stevens et al. present a good example of design, 
although the needs assessment phase, especially on the learners part is not very clear. 

In another example, Card et al. (2011) described the design of an HIV prevention website addressing young 
African American women. The authors specified a theoretical background for the intervention. They translated the 
face-to-face version of the intervention into a multimedia equivalent appropriate for computer-based interventions, 
using the software Flash. Later, measuring sexual and contraceptive behavior changes after the implementation of 
the intervention, they showed positive results on knowledge, self-efficacy, and risk reduction behavior. The authors 
did not report on other design features used in the creation of the intervention, leaving questions to be answered on 
the needs assessment and formative evaluation of the intervention. 

In fact, Kinzie (2005) formulated a set of instructional design strategies in an attempt to offer guidance on 
the design of health education interventions. These strategies drew from several health behavior models, in addition 
to Gagne’s nine events of instruction. However, after conducting a review of literature on adolescent smoking 
prevention intervention, she reported an inconsistency in the use of these strategies, pointing to the difficulties 
associated with comparing outcomes of interventions and proving their effectiveness. 

As for the studies on mhealth interventions, they reveal that the educational emphasis is low in mobile 
applications (Chomutare, Fernandez-Luque, Arsand, & Hartvigsen, 2011). These interventions lack focus on the 
instructional message, although they provide advantages in tailoring, feedback, and support (Fukuoka et al., 2011; 
Whittaker et al., 2011). 

Hence, there are efforts invested by health professionals involved in the creation of ehealth and mhealth 
interventions to address the instructional design process of the interventions. However, these efforts are not always 
rigorous and are do not always follow a thorough systematic approach, which can jeopardize their efforts in reaching 
their goals. 

Therefore, the purpose of this study was to explore how health professionals create ehealth and mhealth 
education interventions to get a better understanding of the framework through which they analyze, design, develop, 
implement, and assess the interventions, in addition to the manner in which they use learning theories, instructional 
models, and elearning design principles. This paper will focus on two of the research questions explored in the 
study: 

1 . How do health professionals use theories and models from the field of education to create ehealth and 
mhealth education interventions? 

2. How do health professionals use principles of elearning and mlearning design to create ehealth and 
mhealth education interventions? 

Methods 

The study followed a qualitative approach of inquiry. A purposeful sample of 12 participants was reached, 
using criterion and snowballing sampling strategies (Miles & Huberman, 1994). The data collection methods 
consisted of interviews, planning materials, and artifacts. The interviews were analyzed using the constant 
comparison method (Glaser & Strauss, 1967) and the planning materials and artifacts were analyzed for 
corroboration of evidence from the interviews. Rigor and trustworthiness were achieved through triangulation, 
member checking, audit trail, and peer debriefing. 
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Findings 


The profiles of the participants showed variation in their characteristics. Their ages ranged from 25 to 61 
years. Their years of experience in ehealth and/or mhealth education interventions varied from 2 to 17 years. Their 
academic background varied between health related sciences, psychology, systems design, and instructional 
technology. The ehealth and mhealth interventions of these participants varied in topics to cover nutrition, prenatal 
care, diabetes, smoking, mental health, cancer prevention, sexual and reproductive health, and general health. The 
interventions targeted populations in the USA, Africa, Asia, Australia, and Europe, ranging between school-aged 
children, teenagers, and adults. 

The accounts of the participants also revealed a wide variability in the way the health education 
professionals create their ehealth or mhealth education interventions. Due to this variability, a descriptive approach 
was followed to present the instructional design components, followed by a discussion on the themes emerging from 
the findings on research questions. 

Description of the instructional design components 

Overall, the participants highlighted three phases of instructional design: analysis, design, and evaluation. 
However, none of the participants used the systematic approach recommended in the instructional design models 
available in the literature (see Gustafson & Branch, 2002). Nonetheless, in their accounts, the participants 
elaborated on all components of the ADDIE framework, particularly on the analysis, design, and evaluation phases. 

Analysis. This phase of instructional design includes a needs assessment that explores the nature of the 
performance problem. In health education, the performance problem is the health behavior. Therefore, this section of 
the findings reflects how the participants assessed the health behavior problem and how they identified their goals 
for designing a health education intervention that will help solve the problem. Four types of needs assessment were 
revealed: (1) choice of health behavior problem and related target audience, (2) assessment of health behavior needs, 
(3) determination of learning outcomes, and (4) learner’s analysis. 

Choice of the health behavior problem and the related target audience. Four patterns emerged on how 
the participants selected the health behavior problem and the related target populations: (1) choice reflected the 
mission of the organization, such as William who focused on sexual health, the health issue his organization works 
on; (2) choice reflected the area of expertise or interest of the participants, such as Daniel who focused on addiction 
because this was his area of expertise (3) choice reflected the need of a stakeholder, such as Emily who addressed 
the issue of prenatal care in pregnant teenager girls because this was the need voiced by the clinic administrators; 
and (4) choice was dictated by a grant, such as Leah who addressed topics funded by grants she received. 

The participants therefore, showed variability in the choice of the targeted health behavior problem and 
related populations based on to the nature of their organization and the different stakeholders. 

Assessment of health behavior needs. The selection of the health behavior problem was followed by a 
more elaborate needs assessment in order to address the health behavior needs of the target population. All the 
participants conducted this phase of the analysis but they used four different methods: data collection, expressed 
need, felt need, and review of literature. 

Data collection. Ten of the participants measured the need through the collection of data directly from the 
target population. Depending on the resources available to the participant, the data collection was conducted 
through surveys or health assessment tools embedded in the interventions. 

Expressed need. Expressed needs are needs voiced by the target population (Morrison, Ross, & Kemp, 
2007). In this study, the participants listened to the expressed needs of the target population through focus group 
discussions and informal conversations. 

For example, explaining the needs assessment for the mhealth smoking campaign Leah said, “In our focus 
groups. . .we found that young adults were particularly skeptical about pharmacotherapy that they sort of felt like 
they shouldn't need it. . . that they sort of be able to will it through these types of things.” These participants 
discussed with their learners their needs in order to get a better understanding of them. 

Felt need. Felt needs are desires for improvement felt by the learner or the expert (Morrison et al., 2007). 
Here, the participants identified a felt need from the subject-matter expert, or in this study the health practitioner, 
who’s input stemmed from experiences with the target population. For example, Isabella described how the need 
was assessed based on the felt need of the health workers. She said, “In all regions there was a demand from the 
healthcare workers to set up a mobile project aiming at enhancing the diabetes awareness in the diabetes patients and 
from there onwards Through felt needs, the participants obtained a better perspective on the aspects health 
problems that needed to be addressed. 
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Review of literature. All the participants conducted extensive reviews of literature. They looked at 
research that reported trends and issues related to the health behavior problem, its corresponding health behaviors, 
and the successes and failures of the approaches taken in addressing them. 

In identifying the health behavior needs of their populations, most of the participants did not exclusively 
use one single method of data collection, but they used a mixed method approach. William summarized it in one 
statement: 

Like I said the first step of the course is understanding what we want to do and doing formative 

research understand what the communities needs are and so we do some literature review and then 

we do some focus groups, survey interviews, whatever is required for that specific project. 

The emerging patterns of the health behavior needs assessment are congruent with the categories of needs 
described by Morrison et al. (2007) who identify normative and comparative needs from data collection and review 
of literature, expressed needs from the target population, and felt needs from the professionals. Additionally, the fact 
that most of the participants used more than one method to assess the health behavior needs reflect an investment in 
efforts to understand the factors surrounding the health behavior problem in order to address it appropriately. 

Determination of learning outcomes. All the participants created interventions aimed at the improvement 
of health. However, these interventions varied in the targeted learning outcomes. Three learning outcomes that 
emerged from the data are change in knowledge, in behavioral skills, and in attitude. The participants sometimes 
explicitly stated their desired learning outcomes and other times their accounts reflected what they wanted to 
achieve from their interventions. 

Overall, the choice of the learning outcomes was influenced by challenges and limitations perceived by the 
participants in achieving them. For example Anna said, “Achieving human behavior is very difficult. ... It is a very 
difficult challenge that we are facing, so we are just trying to be one little slice of the big picture puzzle.” Robert 
also stressed the limitations by saying, “ It really requires a longer engagement. . . and how do we do that in way that 
is still sufficiently lean in terms of resources and time. . .. It’s going to be a challenge.” 

However, the learning outcome patterns reflected to a certain degree the domains of learning used in 
instructional design and described in the literature. Morrison et al. (2007) discuss the cognitive domain 
(knowledge), the affective domain (attitudes) and the psychomotor domain (more or less behavioral skills). Dick, 
Carey, and Carey (2009) and Smith and Ragan (2005) refer to Gagne’s types of learning outcomes (i.e., declarative 
knowledge, intellectual knowledge, cognitive strategies, attitudes, and psychomotor skills). 

Although the participants did not reveal a systematic approach of determination of learning outcomes 
described in instructional design literature, nevertheless, they were cognizant of the different domains, and they 
attempted to cover them to the extent that it was possible for them. 

Learner’s analysis. Learner’s analysis identifies the characteristics of the learners that need to be 
considered in order to maximize the impact of the intervention (Dick et al., 2009; Morrison et al., 2007). In this 
regard, the participants seemed to give great importance to learner’s analysis, revealing four patterns: (1) assessing 
the technology needs and preferences of the learners, (2) assessing the learner’s learning preferences, (3) assessing 
the health behavior characteristics of the learner, and (4) assessing the design preferences. 

Assessing the technology’ needs of the target population. Three participants discussed the assessment of 
technology needs of their target population revealing a variation in how they did it. For example, Emily discussed 
with her team members how a mobile intervention would be well received by at-risk pregnant teens because of their 
age group. Anna expressed that since children enjoy playing video games on their computers, they will be 
motivated to explore her ehealth education interventions. These participants revealed how they catered to the 
technology preferences of their learners in order to engage them. 

Assessing the learners ’ learning preferences. Six of the participants depicted learning preferences among 
their learners depending on age, type of technology, or learning approaches. For example, Mia and Robert 
explained how learners engage in different learning approaches. Mia said, “Those [game] mechanics are really a 
hook to keep their attention and their motivation going” and Robert said, “It was found that active learning, hands- 
on learning and self-directed tended to show higher level of engagement.” Moreover, William and Robert revealed 
their understanding of how people learn differently and through different modalities and considered this 
characteristic in planning their interventions. 

Assessing the health behavior characteristics of the learner. Seven participants discussed the health 
behavior characteristics of their target audience. Their assessments were mostly based on perceptions and 
experiences with the target audience such as lack the skill of self-motivation, inaccessibility to health care services, 
and lack of analytical skills in reading health information. Again, the participants showed efforts in gaining an 
understanding of the health behavior characteristics of the learners in order to answer to their needs in the 
interventions. 
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Assessing the design preferences. Five participants explained how their approach to the design of the 
intervention is based upon discussions and feedback from the target audience. They felt that the more the 
intervention suited the preferences of the target population, the more it engaged them. Leah, Sophie and William are 
examples. 

Leah talked about the opinion leaders that she consulted: 

We conducted a massive quantitative survey primarily to identify what we called youth opinion 
leaders. . . so these are kids that were sort of the most popular kids in each class. . .they can help us 
develop this thing so that when it was done it would be interesting to the popular kids and 
hopefully then they could sort of lead the opinions of others. 

Sophie explained the kind of feedback she collects from students: 

I’m actually getting feedback and assessment from the students. . . finding out what is it they want 
to see so we can tailor the website and make certain that that’s exactly what they want. 

William summarized it all by saying: “So the principle that we work on is, is that, you know, we don’t know best.” 
Here also, the participants revealed awareness towards the need to respond to their learner’s design preferences. 

Overall, the participants depicted a wide range of elements in their learner’s analysis. They touched on 
technology, learning, health behavior, and design preferences, which are all essential in the analysis phase of 
instructional design. The instructional design literature varies in process and elements assessed in the analysis 
phase. However, any information that adds to the understanding of the learner will help in the success of the 
instruction (Dick et ah, 2009; Morrison et al., 2007; Smith & Ragan, 2005), in this case the health education 
intervention. 

In summary, all the participants conducted the analysis phase of ADDIE, though to varying degrees. They 
defined a target health behavior problem and the related populations based on the nature of their organization and 
the different stakeholders. They clearly identified the health behavior needs of their populations and analyzed their 
learners, though with different methods. As a result, they defined appropriate learning outcomes that they perceived 
attainable and carried out a goal analysis to achieve the learning outcomes. 

Design. This phase of instructional design includes defining the objectives, the learning activities, and the 
media to be used for the instruction. In this study, participants emphasized three design aspects: design process, 
content design, and learning activities design. 

Design process. All of the participants discussed the process of their designs, revealing three approaches: 
multidisciplinary approach, expert approach, and learner participation approach. 

Multidisciplinary’ approach. A multidisciplinary approach to design involves the collaboration of several 
team members who are skilled in different disciplines. The team members can be instructional designers, content 
specialists, education specialists, and production specialists. This approach has the potential of maximizing the use 
of the resources needed in creating a learning intervention (Brooke, Bell, & Oppenheimer, 1976; Care & Scanlan, 
2001). In this study, four participants working within large teams showed a multidisciplinary approach to design. 
For example Mia reported,” The design and engineering team . . . brainstorm some ideas on how to best implement 
. . . [the] goals that we have expressed earlier in the first design meeting.” William also said, “One person will 
become the project lead and that person then gets together a team for implementing the design and implementing the 
project. . . market and communication team. . . our graphic designers and our engineers.” 

These participants using the multidisciplinary approach revealed how they benefited from the expertise and 
opinions of the different team members to facilitate the design of their interventions. 

Expert approach. An expert approach to design indicates the subject-matter expert taking on the tasks of 
the instructional design process. Although subject-matter experts are highly knowledgeable on the content, they are 
not as skilled in how to transform the content into learning materials that address its objectives (Dick et al., 2009; 
Lynch & Roecker, 2007). In this study, four participants working in smaller teams approached their design from the 
perspective of their area of expertise. For example, for Daniel, who is a clinical psychologist and expert on 
behavioral change, his design was heavily based on strategies of behavioral change. He said: 

1 think the interventions. . . have to be just time interventions. . . . This is where the users. . . answer 
a question and then you adapt based on where they are in the moment. . . Based on the behavioral 
change literature this is why you want to change, what would you say to yourself if you were 
considering not changing, what are your new behaviors? 

Similarly, for Ryan, who is an expert in medical and health issues, the design leaned towards providing assessment 
and informative advice on health problems. 

A lot of what we do is based on. .. a family medicine approach. ... Let’s say you take a checkup on 
your risk factors from heart disease and you tell me that you have a history of high blood pressure, 
you have a history of high cholesterol, and you don’t exercise much. What our system actually 
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does is it actually starts . . . feeding the database with that information which will then allow us to 
target you with personalized information. 

So, these participants played the double role of the subject-matter expert and the instructional designer. 

This role was influenced by their expertise in the health and behavioral change domains and their intuitive approach 
to design. 

Learner participation approach. A learner participation approach, also referred to in the literature as 
learner-centered or user-centered approach, aims at involving the learner in the early stages of design in order to 
enhance the achievement of the objectives of the learning material (Corry, Frick, & Hansen, 1997; Vincini, 2001; 
Zaharias & Poulymenakou, 2006). In this study, a learner participation approach was used where members of the 
target population were involved in the design process from the beginning. Lillian exemplified this approach the 
most. She explained: 

So, we use what’s called user-led design. ... Young people. . . have an enormous say in what the 
website looks like. So as part of the participatory design process, you work with young people to 
capture the elements that they think should be on the website. . . . [They] bring in a list of websites 
that they really like, that they find engaging and they might go well 1 really like the font on the 
Facebook site, but I really the images on YouTube or I really like the MTV website, has excellent 
edge, elements on their homepage. . . so they’re never starting from the blank slide. 

William also used the learner participation approach through focus groups with his target population. It is 
important to note that William used the multidisciplinary approach as well, showing that multidisciplinary and 
expert approaches have to be exclusive from each other; however, learner participation approach can be inclusive 
to both. 

So, Lillian and William revealed how they reach out to their target population to collaborate with them on 
decisions regarding content and activities, in order to create an intervention that fits their needs. In collaborating 
with their learners as such, they ensured their engagement and they increased the chances of achieving the desired 
learning outcomes. 

The participants used different design processes as they worked on their interventions. This was 
determined by the nature of the organization they were part of and the resources available to them. The 
multidisciplinary approach maximized the use of resources; the expert approach was influenced by the area of 
expertise of the participant; the learner participation approach centered on including the learner in several phases of 
the creation of the intervention. 

Content design. All the participants discussed how they chose the content for their interventions. 
Consequently, three approaches emerged: subject-matter expert approach, collaboration with learner approach, and a 
mix of collaboration and subject-matter expert approach. 

Subject-matter expert approach. Six participants, exemplified how subject-matter experts, such as 
physicians, nurses, or dietitians, selected and finalized the content based on their knowledge and understanding of 
the health behavior needs of the target population. For example, Anna said, “The dietitians will set the objectives 
that for an age group this is normally what we wanted to teach them.” Isabella reported, “ [The physician] did 
provide her health knowledge together with some diabetes experts.” More systematically, Leah said, “[The health 
communication specialist] put together ... an Excel sheet to basically identify different types of messages so 
cognitive restructuring, encouragement, all these behavioral skills.” 

These participants relied on the expertise and knowledge of the subject-matter experts to build the content 
of their interventions, ensuring its accuracy. 

Collaboration with learner approach. The two participants who exemplified the collaboration with the 
target population were Lillian and William. These are the same participants who showed a learner participation 
approach on the design process earlier. 

Lillian: A draft and a fact sheet might be written by a clinician. . . then that fact sheet is actually 
shown to a group of young people who looks at it and provides feedback on the way that 
the language it used, the way that it set out, how it actually presents the content, and then 
the fact sheet is revised. 

William: We did some focus group discussions with the community, with young people and that 
helped us design and develop specific messages. . . so these messages came from the 
young people. ... We went out and we asked, “Hey, what are the issues that you want to 
learn about and what are some of the tips that you would like to get or share with other 
young people? ... We cut those tips and then we turned them into text messages. 

Lillian and William recruited the help of their learners in building the content to make sure that it answered 
their needs. Their approach revealed their belief in the value of partnering with the learners all aspects of the design. 
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Mix of collaboration and subject-matter expert approach. Daniel was the only participant who used a 
mixed approach towards the design of the content of his intervention. He said: 

The key to this program is that people write their own messages. . . we tried to guide people 
through a few questions. . . What we found is that people weren’t writing the messages so. . .we 
said, “Okay, why don’t we do both? We’ll write the messages for people just like 50 or 60 
messages” so that will compliment the messages people write to each other. 

Daniel wanted his learners to collaborate on the content of the messages. At the same time, he found that 
he needed to develop some of them from his perspective as an expert. Here, he supported his learners with expert- 
oriented content. At the same time, he motivated his learners in taking ownership of their own messages. 

Learning activities design. All of the participants revealed one or more learning activities through which 
they delivered their content. The most common activities were: text messages, multimedia, interactive applications, 
and resource centers. 

Text messages. Six of the participants in this study used text-messages for their mhealth education 
interventions. However, these text messages varied in types. For example, Isabella talked about simple text 
messages with no links to the web because of barriers of connectivity. She also used a two-way communication 
because her learners had to supply their diabetes educators with certain information and receive feedback 
accordingly. Similarly, Leah’s intervention needed an interaction between learner and learner, “We had a 
component called text buddy. ... One person would be paired with another person. We had instructions on the web 
site. ... how you sent messages to each other.” 

On the other hand, Daniel used a one-way communication to avoid dealing with legal issues, “We don’t 
[do] interactive messaging for our programs. . .because of the FDA rules. . . once you are going beyond 
information. . . you can make the argument that you are doing an intervention.” 

Therefore, even with a learning activity that is as simple as text-messages, there were variations: text- 
message only, text-messages with links to the web, one-way communication, and two-way communication. The 
selection of the variation depended on reasons such as Internet connectivity, regulations, and the desired learner’s 
activity. 

Multimedia. Four participants used videos for their ehealth education interventions. Anna used a basketball 
game where children play against junk food bandits. Robert developed videos presented by experts on cancer, Ryan 
directed his learners to open license YouTube videos, and William developed his own YouTube videos. 

So, these participants used videos either to house a game or to present additional resources on the health 
behavior problem. As it was evident on their websites, these videos included animations, scenarios and subject- 
matter expert presentations that aimed at engaging the learner and presenting the learning material in multiple ways. 

Interactive applications. Several participants in this study used applications that required the learner to 
interact with them. 

• Anna’s website included several games where, for example, learners learned how to build a healthy 
meal or read food labels. 

• Leah used animated, color-coded frogs to reinforce the learning in her scenario-based modules. 

• Robert created an interactive 3D model of the lungs to provide his learners with an authentic 
experience of healthy and smokers’ lung. 

• Ryan created online check-ups for his learners such as Body Mass Index and calcium calculators that 
were followed by tailored health information. 

• Mia explained about the game mechanics on her website where learners accumulate points that are 
visualized through a plant that grows as more points are earned. 

• William created an ecard partner notification partner system for sexually transmitted diseases (STD), 
where diagnosed teenagers send ecards to their partners to inform them of possible exposure to STDs, 
so they can in turn go and get tested. 

These participants were keen on delivering their learning materials through applications that motivated 
their learners and actively involved them in the learning process. 

Resource centers. Five of the participants created resource centers for their learners either in the form of 
services or in the form of additional information materials, hoping to provide them with all the support they needed 
to achieve the desired health behavior. For example, Anna had over a thousand print materials to help the nutrition 
teachers. Camilla shared health websites with her learners. Lillian created “an online hub where young people can 
download different tools or applications.” William’s website provided “young gay men. . .with STD information and 
HIV information. . .the opportunity to do a self-assessment. . . took them to local testing sites. . . the ability for people 
to have their lab slips directly printed out or emailed from the website.” 
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The learning activities thus described, reveal the creativity and efforts invested by the participants to enrich 
the learning experiences of their learners. The nature of the learning activity was influenced by the type of 
technology used, connectivity issues of the target population, and limitations in resources. 

Therefore, the participants designed their interventions either by working with specialists or subject-matter 
experts, or by collaborating with their learners. They also employed a variety of learning activities that suited their 
target audience and stayed within the limit of their resources. 

Evaluation. The evaluation phase includes formative and summative evaluation (Dick et al., 2009; 
Gustafson & Branch, 2007; Morrison et ah, 2007; Smith & Ragan, 2005). Through formative evaluation, 
instructional designers seek to test their interventions before release to the wider target audience in order to make 
adjustments and corrections. Through summative evaluation, they seek to measure the success of their interventions 
in achieving the intended objectives. Here also, the participants showed variability in conducting evaluation. Eleven 
of the participants conducted formative evaluation and all conducted some type of summative evaluation. 

Formative evaluation. The 1 1 participants who did formative evaluation on their interventions described 
the purpose of their formative evaluation and its process. The purpose of the formative evaluation was to measure 
attitudes, content comprehension, and usability. For attitudes, the participants looked at whether their learners liked 
the design of the intervention. For example, Anna said, “We do a lot of analytics evaluating which areas of the 
website are the most popular and. . . the level of engagement with the website”. For content comprehension, the 
participants looked at whether their learners understood the content of the intervention. For example, Ryan said, “ 
[We wanted] to see whether the content makes sense.” As for usability (Nielsen, 1993), the participants looked at the 
difficulties encountered by the learners as they moved around the intervention. Again, Anna explained, “We watch 
and see. . .if [the children] know what to do and how to maneuver through the game.” 

Describing the process of formative evaluation, the participants used feedback from their learners, such as 
Daniel who said, “We’ve asked people to sense like, ‘What has been your favorite message?’ that kind of thing.” 
They also used analytics as Ryan described, “I can actually watch in real time a person move through the site and I 
can see where they are pausing. . .we can actually see a heat map so where they mostly likely to click.” In addition, 
they field tested their interventions using focus groups where they discussed with their learners the improvements 
needed to the interventions. For example, Lillian explained, “So you might release the mobile app to a small group 
of young people initially and get them to actually test the application before it’s made more widely available.” One 
participant, Mia, used AB testing to compare two versions of a webpage in order to check which one works better 
for the user (Swanson, 2011). Finally, for some participants, formative evaluation was done through the maintenance 
of the website after launching as Sophie explained, “The reality of it is it’s going to be trial and error. I’m going to 
put this site up. . . then do an assessment. . . basically is it working?” 

Therefore, the participants conducting formative evaluation looked at whether their interventions appealed 
to their learners, whether the content was comprehensible, and whether the learners felt comfortable navigating 
through the interventions. They assessed these elements with various methods and used the results to improve their 
interventions. 

Summative evaluation. All of the participants in the study performed some type of summative evaluation. 
In their accounts, they described the focus of their summative evaluation, its process, and the period over which it 
extended. The focus of the summative evaluation included measuring changes in knowledge and attitudes; changes 
in health behavior, such as Emily and William who tracked the change in health services use as a result of their 
interventions; and the learner’s engagement and usability of the intervention over extended period of time measured 
through testimonials and retention rates. The process of summative evaluation discussed by four of the participants 
only, included research-oriented approaches showing a desire to have empirical evidence on the impact of their 
interventions. The period of the summative evaluation ranged from continuous monitoring of the feedback from the 
learners and the behavioral interactions with the interventions to the more research-based approach through short 
term evaluation at the completion of the intervention or long-term evaluation that went from four weeks after the 
completion of the intervention, to 3-6 months later, to yearly, and even to a five year period. 

Hence, the participants did conduct a type of summative evaluation with different levels of rigor and 
extending over different period of times. They also aimed at measuring changes in knowledge, attitudes, health 
behavior, and they kept tracking the usage and engagement of their interventions by the target audiences. 

Overall, in conducting formative and summative evaluation, the participants revealed the value they place 
on having effective interventions that are liked and understood by their audiences. Even when resources were 
limited, the participants made the effort to gather indicators to assess their interventions. However, the type of 
evaluation implemented by the participants was influenced by their academic background and the resources 
available to them. 
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So, not only did the participants measure the impact of their interventions, they also tracked the level of 
engagement of their learners in order to assess their level of interest and motivation. 

Health professionals use of education theories and models 

On how health professionals use theories and models from the field of education to create their 
interventions, the findings revealed that none of the participants used a specific learning theory or an instructional 
model. However, the participants discussed the learning approaches and the instructional strategies they 
incorporated in their interventions. Consequently, four themes emerged: connections to behaviorist approaches to 
learning, connections to cognitivist approaches to learning, and connections to constructivist approaches to learning. 

Connections to behaviorist approaches to learning. Here, the participants indicated the use of a 
behaviorist learning activity in their interventions by offering some type of reinforcement to the learning process. 

For example, Anna and Leah offered points for goals achieved or skills learned. Mia also offered points, badges, and 
access to a premium version for her intervention when learners achieved their goals. It is important to note that most 
of the five participants, who used these behaviorist techniques, also used constructivist instructional strategies. They 
blended techniques and activities from more than one theory in order to optimize the learning experience of their 
learners. 

Connections to cognitivist approaches to learning. Here, the participants talked about the importance of 
using an instructional strategy that controls the amount of information presented to the learner. Although the 
participants did not mention cognitive load in specific terms (Sweller & Chandler, 1991; Sweller et al., 1998), they 
were very much aware of the issues involved with it. For example, Emily said about the text messages in her 
intervention, “So, we went through that content development. We wanted it to be light. ... we didn’t want to 
inundate the patient.” Likewise, Lillian related what her learners needed, “I guess the feedback that often comes 
from young people is that they want that content to be split into smaller chunks.” Therefore, in designing their 
interventions, the participants avoided long texts that required a lot of heavy reading; they did not provide too many 
choices that created a sense of loss in the learner; and they simplified and chunked the information presented. 

The participants’ emphasis on cognitive load is important to note. It indicates a step in the direction of 
sound instructional design that could be strengthened with a deeper understanding of how learning materials have to 
be structured, organized, and sequenced to facilitate learning (Ertmer & Newby, 1993). 

Connections to constructivist approaches to learning. The participants discussed instructional 
strategies that showed similarities to a constructivist approach to learning, where learning is student-centered and 
knowledge is constructed with multiple perspectives and with multiple representations and within authentic 
experiences (Duffy & Cunningham, 2005; Jonassen et al., 2007). These instructional strategies were problem- 
solving, learning by doing, active learning, authentic experiences, and goal setting. For example, Anna talked about 
decision-making through nutrition education games, such as trying to create a healthy recipe for pancakes, Mia 
specified a do-learn approach for her intervention, and Robert discussed his choice of activities that aimed at 
creating an active learning experience for the learners. Although they did not name specific constructivist 
instructional models described in the literature, the participants designed learning activities that aligned with the 
basic tenets of the constructivist approach to learning. Jonassen (1999) explains that constructivist learning must 
first focus on “a problem, a project, or a question” (p. 217) that drives the learning. The problem, project, or 
question must be contextualized, engaging, and motivating. In this study, the instructional strategies chosen by the 
participants included problems or questions their learners must solve. In addition, these activities were 
contextualized in authentic settings; they were engaging, and motivating. 

In their description of the instructional strategies of their interventions, the participants showed a preference 
to constructivist approaches to learning because of the detailed information they provided in this regard. They also 
emphasized the importance of cognitive load. In addition, they added, though sparingly, behaviorist techniques in 
their interventions. Although the instructional and learning approaches the participants described connect to 
learning theories, the participants did not intentionally use a learning theory nor did they choose an instructional 
model. The findings suggest that the participants did not take a scholarly approach in looking at the learning theories 
and instructional model while creating their interventions. However, they invested efforts in creating learning 
activities that reflected instructional models of different learning theories. This focus on instructional strategies 
reflects a genuine effort in creating successful learning experiences, which if embedded in instructional models and 
framed within learning theories that have been researched in the literature, could facilitate the design process of the 
interventions and yield better learning outcomes. 
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Health professionals use of elearning design principles 

On how health professionals use principles of elearning and mlearning design to create their interventions, 
the participants showed an extensive use of essential elearning design principles that are portrayed in the literature 
(Alessi & Trollip, 2001; Hill et al., 2004; Moore, 1989). In this regard, seven patterns emerged: learner-content, 
learner-expert, and learner-learner interaction; learner control over navigation, provision of help and resources, use 
of multimedia, engagement, user friendliness, and visual appeal. However, these participants applied elements of 
elearning design in varying degrees depending on the availability of resources of manpower and time to them. Once 
again, they showed an application of good elearning design, without necessarily being driven by the research-based 
principles that lie behind it. 


Conclusion 

The description of the process through which the participants created their interventions revealed their 
incorporation of essential elements of instructional design described in the literature. Although they did not follow 
specific instructional design models, they elaborated on the phases of analysis, design, and evaluation. York and 
Ertmer (2011) posit that even seasoned instructional designers do not adhere to specific instructional design models, 
but use instructional design heuristics based on experiences in the field. Similarly, Visscher-Voerman and 
Gustafson (2004) reported that experienced instructional designers using ADDIE, deviate from it and implement it 
in different ways. Interestingly, none of these participants in this study was an instructional designer, which explains 
why none of them referred to the use of a specific instructional design model. However, they did seem to employ a 
heuristic approach in the design of their intervention. This is a noteworthy finding for non-instiuctional designers 
working in the field of health education attempting to design instruction without training, and it is one worth 
investigating in order to understand the factors behind it. On the other hand, the fact that the participants leaned 
towards constructivist and cognitivist approaches in their interventions is a promising finding for the field of health 
education. Historically, health education focused on the transmission of knowledge through a teacher-centered 
approach (Keyser & Broadbear, 2010). However, scholars in the field voiced concerns over the need to teach 
thinking skills and to support the learner in the decision-making process of behavioral change, to shift the learning 
paradigm of health education towards a more cognitive -based and collaborative approach (Greenberg, 2010; Keyser 
& Broadbear, 2010; Welle et al., 2010). Therefore, these participants showed an alignment with the hopes and 
recommendations of health education scholars for the paradigm shift in health education. 

Implications for Practice 

Novice instructional designers, as our participants are, need the guidance of instructional design models 
until they become experienced in the process, in order to get the foundation in knowledge and skills (Dick, 1996; 
Nichols, 1995; York & Ertmer, 2011). Lillian talked about developing a guidebook for central design for her project 
partners. Similar to her idea, a guidebook can be developed to serve as a reference for all health professionals. As a 
group, the participants showed a design process that covered the basic components of instructional design. 

However, individually, they showed a great level of variability in the methods and the extent to which they designed 
their interventions, based on their academic background, type and context of interventions, and the resources 
available to them. Since none of the participants is a trained instructional designer, they and similar health 
professionals would benefit from the development of an instructional design guidebook tailored specifically to them 
to enable them to create their ehealth and mhealth education interventions more effectively (Gustafson & Branch, 
2002). This is especially important, because the literature on instructional design is mostly focused on academic and 
workforce settings (Reiser, 2007) and the literature on health behavior and health education covers only occasionally 
aspects of learning and rarely the instructional design process (Bartholomew, Parcel, Kok, Gottlieb, & Frenandez, 
2011; Card et al., 2011; Kinzie, 2005; Stevens et al., 2008). 

Moreover, health professionals would also benefit from a repository for health education learning objects. 
Wiley (2000) defines a learning object as “any digital resource that can be reused to support learning" (p.7). By 
following certain technical standards, these digital resources can be reused across multiple electronic platforms as 
sharable content objects (Lehman, 2007; Reiser, 2007). Consequently, such a repository would prevent duplication 
of efforts, increase efficiency, give health professionals visibility in the field, and provide other health professionals 
with limited resources access to learning objects that they can use in their interventions. 

Additionally, health professionals creating ehealth and mhealth interventions can gain from the 
establishment of a community of practice to overcome challenges and learn from each other’s experiences (Wenger, 
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2009). This is specifically meaningful for health professionals who work in small teams and can benefit greatly 
from health professionals working in multidisciplinary teams. 


Implications for further research 

Investigating the decision-making process of health professionals as they create their interventions can shed 
the light on the factors behind their heuristic approach, even without training in instructional design (York & Ertmer, 
2011). Furthermore, the variability of the design approaches of the participants warrants further examination in 
order to explain the impact of the variability on the quality and effectiveness of the interventions. Additionally, 
comparative studies can be conducted between mhealth and ehealth interventions to shed the light on similarities 
and differences in the design process and effect on health behavior. 
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Introduction 

The Human Resource Management and Development (HRM & D) literatures offer several strategies for 
mapping employee learning to business performance, including focusing on learning transfer (Brinkerhoff & 

Apking, 2001), designing instructional interventions with specific business, performance and learning objectives in 
mind (Hodges, 2002), ROI models (Phillips, 1994; Phillips, 1997a; Phillips, 1997b) or descriptions of what 
organizations should be doing to ensure that investments in employee learning yield concrete business results (Wick, 
Pollock, & Jefferson, 2010). However, studies vary in terms of how learning and business impact are defined, with 
little consensus about what to measure, whether there is a need for industry-specific measures, and what specific 
measurement methodology to use. Perspectives also vary about whether or not subjective perceptual measures of 
performance can be used where objective accounting-based data do not exist. (Cleary, O'Donnell, O'Regan, & 
Bontis, 2007; Coscarelli, Burk, & Cotter, 1995; Levenson, 2005; Phillips, 2009). 

In this paper we explore how organizations in the professional services sector measure the business 
outcomes of employee learning. First, we review the literature and identify several challenges associated with how 
learning, measurement and business impact are defined and operationalized. Next, we draw insights from one-on- 
one depth interviews with decision-makers responsible for employee learning and development strategies in 15 
professional services firms to assess the adoption of current measurement models in a strategic context. Lastly, we 
suggest a blended approach to measuring the business impact of learning and identify some opportunities for future 
research. 


Models in the Literature 

Table 1 summarizes the wide variety of measurement models established in the field. The models share 
some similarities in that they all seek to provide a roadmap to selecting and tracking the key indicators linking 
employee learning to organizational performance. They are also consistent in acknowledging that different 
organizations have different perspectives on what constitutes “performance”. Nevertheless, there are challenges 
associated with these models, including (a) focus on the training dimension of learning, with little emphasis on other 
forms of formal or informal learning; (b) wide variation in the meaning of “impact”, with working definitions 
ranging from accounting-based indicators to perceptual measures of value, and; (c) limited empirical evidence 
validating the models or the extent to which they have been adopted. 
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Table 1 

Summary’ of Selected Models/Methods of Measuring the Business Impact of Employee Learning 


Authors 


Key Model Concepts 


Working Definitions 


Kirkpatrick (1959, 1996) 
Kirkpatrick & Kirkpatrick 
( 2010 ) 


Reaction-Learning-Behavior-Results 
Return on Expectations (ROE) added 
to four levels 
Focus on training 

program/intervention and its results 


Learning through training; 
potentially applicable to other 
employee development activities 
Impact-business results derived 
from learning interventions as 
measured by (a) on-the-job 
performance/Level 3 and (b) 
quantifiable business results (e.g., 
reduced costs, increased 
production)/Level 4 


Phillips (1994, 1997) 


Extends Kirkpatrick to five levels 
including ROI 

Ten techniques to isolate the effects of 
training vs. other sources of business 
improvement 


Learning through training; 
potentially applicable to other 
employee development activities 
Impact-economic profitability, 
specifically accounting-based ROI 
via perceived monetary values 


Kraiger (2002) 

• Assesses the training program, 

• Learning through training; 


changes in the learner and changes in 

potentially applicable to other 


the organization 

employee development activities 


• Focus on intended purpose for 

• Impact-outcomes at the 


evaluation (decision making, 

intervention, learner and 


marketing, feedback to participants. 

organizational level as measured by 


instructors, instructional designers) 

trainee reactions 


Brinkerhoff (2003) 


Success Case Method focusing on 
systems 

Evidence- and story-based 

Focus is the performance management 

system and the role that learning 

played in it to achieve results 

Contextual reasons for training 

transfer/successful vs. unsuccessful 

cases 


Learning through training and the 
interaction of training with 
performance management factors 
Impact-business results derived 
from learning interventions and 
employee performance as measured 
by ROI and statements of benefits 
derived from actual cases, 
verifiable records, and direct 
evidence of business value in 
specific cases of training usage 


Spitzer (2005) 


Predictive-Baseline-Formative-In 
Process-Retrospective LEM model 
Alignment with learning design and 
development phases 


Learning through training 
Impact=business results (financial 
and behavioral based on perceptual 
ratings) derived from priorities of 
organization’s leaders 


Bersin (2008) 


A nine-component model that centers 
around satisfaction, learning, 
adoption, utility, efficiency, 
alignment, attainment, individual 
performance, and organizational 
performance. 

Contribution-Feedback- Activity to 
measure the contribution of informal 
learning 


Learning through training; informal 
learning through structured 
activities (e.g., postings to 
enterprise-supported collaboration 
sites) 

Impact=causal chain of quantitative 
and qualitative indicators 


617 



With all this definitional turbulence, it remains unclear as to what organizations are actually doing to 
measure the business impact of employee learning. Anecdotal evidence suggests that there may be considerable 
variation by industry sector. This study seeks to cut through the definitional turbulence by researching measurement 
practices in the professional services sector using Von Nordenflychf s (2010) classification of professional services 
firms (PSFs): (a) classic PSFs (e.g. law and accounting firms), characterized by a high knowledge intensity, a 
professionalized workforce, and low capital intensity; (b) professional campuses (e.g. hospitals), characterized by a 
high knowledge intensity, a professionalized workforce, and high capital intensity; (c) neo-PSFs (e.g. management 
consultants), characterized by a high knowledge intensity and a low capital intensity, and; (d) technology 
developers (e.g. R&D firms, biotechs), characterized by a high knowledge intensity and a high capital intensity. We 
specifically selected the professional services sector for this study because that is the industry sector in which firms 
have a global impact (Suddaby, Cooper, & Greenwood, 2007), in which human capital is the largest asset 
(Lowendahl, 2005), and in which employees tend to be more likely to participate in work-related learning 
opportunities than employees in other industry sectors (Xiao & Tsang, 2004). Our research draws insights from a 
group of senior FIRM & D decision-makers to explore the people, processes and tools that support employee 
learning and the impact of these on a firm’s performance. 

Method 


Research Design 

Finding no published studies about measuring the impact of employee learning in the professional services 
sector, we deemed a qualitative exploratory approach to be appropriate for this research. An exploratory or content- 
driven study begins with relatively open-ended research questions (e.g., what do PSFs think about employee 
learning?) and often uses participants obtained via purposive sampling whereby a non-representative subset of a 
larger population is recruited to serve a very specific need or purpose (Guest, MacQueen, & Namey, 2012). 
Qualitative exploratory research focuses on inductive discovery whereby the researcher collects data that is 
primarily non-numeric (text, images, audio) and allows trends, themes or ideas to emerge from the data without pre- 
determined analytic categories, codes or theoretical frameworks. The researchers gain knowledge based on 
constructivist perspectives (the multiple meanings of individual experiences, meanings socially and historically 
constructed, with an intent of developing a theory or pattern) or advocacy/participatory perspectives (political, issue- 
oriented, collaborative, or change oriented) or both (Creswell, 2003). In this way, the researchers can generate 
hypotheses for future study or build theoretical models from the data. 

Participant Selection 

Consistent with studies of subjective and objective company performance (Bontis, Crossan, & Flulland, 
2002; Dawes, 1999; Kamath, 2007), our study assumes that decision-makers responsible for an organization's talent 
management and employee development strategies (e.g., VPs of Fluman Resources, Chief Learning Officers, 
Directors of Talent Management) have a level of knowledge - or have access to those with the knowledge - and the 
authority to respond to questions about the organization’s practicies, policies, and company performance. 
Consequently, participants were selected via a purposive sampling of individuals meeting the following criteria: (a) 
job title of director or higher in one of Von Nordenflychf s (2010) four classes of professional services firms (classic 
PSFs, professional campuses, neo-PSFs, technology developers), and; (b) responsibility (direct or through an 
immediate subordinate) for talent management strategies, including the definition of various learning and 
development opportunities. 

A call for participation that described the study’s purpose and participant inclusion criteria was posted on 
Linkedln, the largest online professional network (Linkedln Corporation, 2013). Interested individuals were 
instructed to click on a link to the URL hosting an Informed Consent Form and provided their contact information to 
schedule interviews. Neither of the researchers conducting this study were acquainted with the participants prior to 
the start of this research project. A total of 15 participants who met the inclusion criteria were interviewed for this 
study and included 10 males and 5 females. Although the participants work in a variety of professional services 
firms (PSFs), most of the participants are in the neo-PSF and technology developer sub-sectors and have job titles of 
director-level or higher. The median number of years of participant experience in the field is 19.0 and nearly all of 
the participants hold advanced degrees in either HR-related or education-related disciplines, with some also holding 
the ASTD Certified Professional in Learning and Performance (CPLP) certification. The median firm size was 8,000 
employees. 
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Data Collection, Instrumentation and Analysis 

Our lead researcher conducted one-on-one phone interviews over a 6 month period. Each interview lasted 
30 minutes on average. The open-ended interview protocol, which is grounded in the issues and challenges 
identified in the literature, focused on three broad areas: (a) learning opportunities offered to employees, including 
how learning needs are identified; (b) measuring the impact of learning on the firm’s business, and; (c) perceptions 
of the firm’s responsibility for fostering employee learning and development. All interviews were audio recorded, 
transcribed, and a copy sent to the relevant participant to confirm the accuracy of the transcript's content. 

We deemed applied thematic analysis to be the appropriate data analysis strategy for this study. Applied 
thematic analysis consists of identifying key themes in the transcript text, transforming those themes into codes and 
aggregating them into a codebook (Guest, MacQueen, & Namey, 2012), the process for which entailed three cycles 
of analysis. The first cycle involved descriptive coding, whereby codes or short phrases are created to describe a 
portion of the text’s content and essence for later purposes of pattern detection and categorization (Miles, 

Huberman, & Saldana, 2014). Using the interview protocol as a structural guide for code development, one 
researcher developed a basic set of codes for each of the questions. As the researcher systematically reviewed the 
transcripts to segment and code the text, additional codes emerged while other codes were revised. The researcher 
then began consolidating these codes into patterns or common threads across the participant interviews, the second 
cycle of the process. 

Once a preliminary codebook was developed, the second researcher coded eight randomly selected 
transcripts using the preliminary codebook, uncovering additional patterns and revising some codes. We compared 
our coding results, discussed any discrepancies and the reasons for those discrepancies, agreed on a solution, and 
then collaboratively revised the codebook. As an example of our coding practices, part of a quote from a participant 
stated, “The third piece we do for learning is around technical training, be it on systems or workflows or products”; 
we coded this quote as “technical training.” The third and final cycle involved each researcher independently coding 
eight randomly selected transcripts using the revised codebook. Only two statements were coded dissimilarly by the 
researchers. Following discussion of these statements, we agreed to classify those statements into “Other 
Comments” category and the codebook was modified. Remaining transcripts were examined, with no inter-coder 
incongruities. Given the small sample size (n=15), this subjective assessment was deemed an appropriate method of 
checking inter-coder agreement (ICA), the extent to which two or more researchers code the same qualitative data 
set in the same way (Guest, MacQueen, & Namey, 2012). Table 2 illustrates our coding process. 

Table 2 


Examples from the Codebook 


Employee Learning Opportunities (Interview Q. 1) 


Codes 

Themes 


• Selective leadership development • Formal learning delivered face-to-face or 

• Industry learning virtually 

o Business development excellence 
o Knowledge of market, agency, product 
o Corporate strategic initiatives 

• Technical training 

• Professional development programs 

o Job profile competency courses 
o Career development 
o Professional skills 
o Long-term needs 

• Communities of practice • Informal learning, face-to-face or virtually 

• Structured learning 

o Formal mentoring 
o Job rotation 

o Experiential simulation-based learning 

• Unstructured on the job learning 

o Informal mentoring 
o Project teams 

• Coaching 

• Lunch and learns 


619 




Results 


Learning Opportunities Offered 

Participants in our study appear to associate a learning opportunity with any mechanism in which the firm 
invests resources - financial, technical infrastructure, staff- to support an employee’s ability to acquire skills and 
competencies to improve performance. When asked about learning opportunities offered by the firm, participants 
mentioned a variety of formal and informal learning opportunities. Training courses were the most common type of 
formal learning opportunity mentioned and include face-to-face, online via the firm’s learning management system 
(LMS) and blended formats. Training content addressed topics such as leadership development, industry-specific 
and company-specific content, technical training, professional development programs, compliance training, 
interpersonal skills, and other business-related skills. Some of the participants’ firms have job profile competency 
courses that are specifically designed to fill in any existing knowledge gaps. There are also a variety of incentives - 
tuition/fee reimbursement, time off, etc. - to encourage employee participation in external as well as internal 
learning opportunities. 

Informal learning appears to be strongly encouraged across all of the PSFs represented in this study. 
Participants mentioned a wide range of structured (e.g., job rotation) and unstructured (e.g., informal mentoring) 
opportunities. Mentoring, lunch-and-learns, and communities of practice (CoPs) were cited most often as vehicles 
for facilitating informal learning. For instance, one participant stated: 

The CoP is more than just a way to communicate but rather a way for people to participate in building 
something that’s going to endure and provide for career development opportunities. It’s going to provide 
for evidence-type activities: writing, presenting papers at conferences, things of that nature. It’s going to 
allow for recognition within the firm. 

(Director of Talent Development, Management Consulting sub-sector) 
Responses to the learning opportunities question show some variation within the various professional 
services sub-sectors. Participants from the management consulting sub-sector of Neo-PSFs were similar to the 
participants from the IT services sub-sector of Technology Developer PSFs and the insurance advisory services sub- 
sector of Classical PSFs in investing in learning opportunities for employees targeted for leadership positions. 
Participants from the financial advisory services sub-sector of Classical PSFs and the healthcare services sub-sector 
of Professional Campus PSFs appear to invest in broad-based opportunities for employees at all levels. 

Employee learning needs are identified based on strategic talent management plans, individual performance 
reviews, and employee self-assessment and development goals, all documented in HRM databases. Formal written 
business cases are rarely used to initiate new learning projects and then, only for high-stakes initiatives such as 
leadership development programs. As one participant stated: 

We have a formal business case process for large strategic programs that we invest heavily in. For example, 
we do a twice-a-year residential program for senior leaders who are nominated to be part of this program. 
It’s a combination of leadership skills and company strategy and the faculty for that program is our 
Executive Team. 

(Performance & Talent Development Leader, IT Services sub-sector) 

Measuring the Business Impact of Learning 

The topic of measurement was introduced at two points in the interview. Following the discussion of 
formal learning opportunities offered to employees, participants were asked to think about some of the business 
performance indicators that are commonly tracked in professional services firms and to describe how the 
contribution of employee learning to those indicators is measured. Four typical indicators were given as examples: 
(a) utilization rate or the number of billable hours divided by the total number of working hours; (b) the ratio of 
senior to junior billable staff; (c) gross margin for professional services, and (d) backlog of signed contracts minus 
billings (Wintner, 2006). Participants were unanimous in noting the difficulty in making a direct correlation between 
training and one or more business indicators, citing a variety of factors that can influence those indicators, including 
learning outside of formal training. Moreover, participants specifically shy away from ROI. For instance, one 
participant stated: 

I would not, as the learning executive, want to be in front of them [sic, the auditors and accountants 
employed at the firm] trying to explain the ROI calculations. I would be eaten alive. So that’s why we don’t 
do the Jack Phillips or any of those other financial ROI calculations. But the other reason is that we feel 
that a better indicator for us is what our clients tell us. So that to me has a direct, fairly clear fit between our 
development programs and our client service. We feel that the revenue will follow. 

(Director of Talent Development, Management Consulting sub-sector) 
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Only the two Healthcare Services sub-sector participants in this study stated that their firms were not currently 
measuring the business impact of training. However, this was due to some business process reengineering efforts at 
their respective firms, part of which includes the creation of appropriate metrics to map employee training and 
development to business results. These participants did share the status of their efforts to date. 

It was only at the second introduction of the topic of measurement - during the discussion of informal 
learning opportunities - that participants offered a detailed description of the ways in which they measure the impact 
of learning on business performance. Overall, participants measure the impact of learning by employee 
output/productivity and indirectly link those accomplishments to the firm’s performance. To achieve this, 
participants mentioned using a variety of measures, as shown in Figure 1. 


Extended, Renewed Contracts 



Figure 1 How selected professional services firms (PSFs) 
measure the business impact of employee learning. 


Client-related Factors 

One set of measures involves quantitative measures of success with client engagements: number of 
extended, renewed contracts; number of referrals; percentage increase in the number of Requests for Proposal 
(RFPs); number of successful project completions, and; number of calls for specific areas of expertise. We classified 
this set of measures as Client Satisfaction. Another set involves quantitative measures of success acquiring new 
business: add-on business or additional work on existing contracts; additional business from new clients; proposal 
win rates, and; re -compete win rates or new contracts from existing clients. We classified this set of measures as 
Winning New Business. A third type of measure is used by those participants whose firms perform work for the 
U.S. Federal Government and involves a series of subjective measures (i.e., rating scales) of how well the contractor 
performed against contract requirements. We classified this as Engagement Assessments and, with Client 
Satisfaction and Winning New Business, grouped all of these under Client-related Factors. 

Organizational Factors. 

Our analysis revealed a variety of quantitative and qualitative measures that focus on individual employee 
performance: feedback from supervisors, both informal and formalized in performance reviews; promotions within 
the firm; perceived improvements in productivity; perceived improvements in performance quality; direct 
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observation of service creation or delivery by supervisor or peers; reports of employee sharing knowledge/learning 
with colleagues; learning outcomes of training as measured by Kirkpatrick Levels 1 and 2 surveys; employee scores 
on quizzes/knowledge checks; employee participation in formal and informal learning opportunities, including CoP 
memberships, and; client ratings of individual employees via employee engagement surveys. We classified this set 
of measures as Individual Level. The last set of measures focuses on processes and programs within the firm that 
support employee learning: management fostering a learning culture through regular communication of the 
importance of a knowledgeable workforce; providing tuition assistance, time off and other forms of compensation 
and benefits to encourage employee participation in learning opportunities; systematic use of data collection based 
on Kirkpatrick Levels 3 and 4; use of other assessment tools (e.g., benchmarking standards, balanced scorecard) to 
foster learning, and; employee recognition programs tied to evidence of employee application of learning to work. 
We classified this set of measures as Organizational Level and grouped these along with Individual Level measures 
under Organizational Factors. 

Firm Responsibilities 

The last area of exploration in this study focuses on how professional services firms view their 
responsibilities in terms of employee learning. Specifically, participants were asked to gauge their firms on a 
learning continuum. At one end of the continuum are firms that say employees are responsible for their own learning 
development and at the other end of the continuum are firms that feel it is the firm’s responsibility to develop 
employees throughout their entire career, cradle-to-grave. Fourteen of the 15 study participants placed their firms in 
the middle of the continuum. The rationale offered centered on organizational willingness to support employees by 
providing learning and development opportunities. However, participants stated that it was up to the employee to 
proactively utilize the available resources. For example, two participants stated: 

There’s a very strong emphasis on providing a number of opportunities and participation in these 
opportunities and establishing ways to commend and reward those who successfully participate. Ultimately 
there’s also the recognition that you can’t force someone to better themselves, you can’t force someone to 
attend training that they’re not really interested in. 

(Director, Institutional Human Capital Development, Financial Advisory Services sub-sector) 
There’s a really great on-boarding process, a three to six months process, where we like to push people 
through a set of learning. After six months, up to a year, then it’s up the employee. Come to us, tell us what 
you’d like, you’re in charge of your own education. That’s why I say we’re kind of in the middle, because 
of these different stages. 

(Director, Management Consulting sub-sector) 

Only the participant from the Insurance Advisory Services sub-sector classified his firm as being on the cradle-to- 
grave end of the continuum citing the highly-regulated nature of the sub-sector where regulations vary from state to 
state and thus, require employees to continue learning new things in order for the firm to maintain its competitive 
advantage. 


Discussion 

Using a qualitative exploratory research approach to examine how professional services firms measure the 
business impact of employee learning allowed us to take an industry-specific deep dive into perspectives and 
practices that contribute to an important HR practice area, namely talent development through learning. One of the 
themes emerging from the data is our finding that PSFs invest in a variety of learning opportunities based on both 
employee goals and organizational goals. This finding is consistent with the literature describing learning 
opportunities as an indicator of the alignment of learning and development with business needs. For example, 
Schmidt and Kunzmann (2006) describe alignment of learning and HR as occurring when learning is determined by 
both individual and corporate interests, and when the organization offers relevant learning opportunities to ensure 
sufficient competence. 

Some researchers have characterized organizations that offer a variety of employee learning and 
development opportunities as having a strong learning culture (Clarke, 2005; Hameed & Waheed, 2011). We would 
categorize the professional services firms in this study as having strong learning cultures because they display the 
organizational prerequisites for a learning culture, namely open communication channels (Teare, 1998), strong 
leadership support (Bass, 2000), a reward system (Bahaudin & Shandana, 2010), mutual reinforcement of individual 
development needs and organizational objectives (Wilkinson & Kleiner, 1993), and the promotion of informal 
learning (Rainbird, Munro, & Holly, 2004). Both the employee and management of the participating firms are 
involved in the decision making processes related to talent development, as reflected in the individual development 
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plans, and in the firm’s talent management plan. Moreover, by classifying their firms as being in the middle of the 
learning continuum, participants characterize employees as being accountable for their own learning and 
development, but the firm as committed to employee learning/development by investing in tools and opportunities 
for learning. 

The finding that participants from three of the industry sub-sectors - management consulting, IT services, 
and insurance advisory services - strategically invest in individuals identified as high potential illustrates what 
Leonard (2005) describes as a leadership development, which implicitly operates under the assumption that not all 
individuals are likely to derive equal benefits from a learning intervention. Further, some deem it may be appropriate 
to reserve standard leadership opportunities for elite candidates, who are likely to gain significantly from it and 
thereby impact the organization’s success (Bledsoe & Robinson- Walker, 2010; Leonard 2005). This does not, 
however, suggest that those sub-sectors decline to invest in learning opportunities for employees at all levels, only 
that leadership development is a strategic necessity. 

Another theme emerging from our study is the blending of client-related factors and organizational factors 
used to measure the business impact of employee learning. Moreover, client-related factors in particular align with 
the PSF business objectives. The findings of our study suggest that no single model or metric is deemed to be the 
“gold standard” of business impact measurement. Instead, the PSFs in our study use a variety of quantitative, 
qualitative, objective and subjective measures of business impact, all of which appear to be grounded in business 
metrics common to professional services firms. Using multiple measures enables management to get a better handle 
on how investments in learning, like other investments, contribute to a firm’s overall performance (Quartey, 2012; 
Thatchenkery, 2005). 

Results of this study revealed that almost all of the learning executives self-classified their firms as being in 
the middle of the learning continuum. Correspondingly, Hameed and Waheed (2011) have affirmed that although 
organizations have a responsibility to provide resources and activities for development; the employee is expected to 
engage in self-development through self-directed learning. Thus, common to both researchers and practitioners 
appears to be the notion that employees are held accountable for their learning and the organization demonstrates its 
commitment by supplying tools and opportunities for learning. 

Implications for Practitioners 

The findings of this study suggest that decision-makers responsible for a firm’s talent management and 
employee development strategies need to step back from the notion of a “best measurement model” and focus on the 
measures and metrics that the firm’s leadership deems to be important. The key to getting the appropriate metrics 
and measures in place and disseminated as broadly as possible is to (a) get the firm’s leadership involved in how the 
learning measures and metrics are linked to firm strategy and have the leadership serve as learning champions; (b) 
visually represent the contribution of learning via charts, graphs, and diagrams that show employees how learning 
contributes to firm performance; (c) keep the metrics simple for ease of understanding and use in action planning 
(Bine, 2002). As such, the contribution of learning becomes part of an overall business metrics program that 
considers not only learning investments but also how the business uses learning. It is an evolutionary, iterative 
process of aligning and realigning the firm’s business strategy, HRM & D strategy, and other HRM functions. 

There is no lack of evidence that firms are not capturing measures that show what the leadership values. 

For example, some studies argue that C-level decision-makers have not been presented with meaningful evidence 
about the positive relationship between learning activities and business success (O’Leonard, 2013: Short & Harris, 
2010), while others cite the application of a one-size-fits-all approach to measurement for all learning and 
development programs, regardless of the impact of these programs on the business or the industry sector in which 
the business operates (Barnett & Mattox II, 2010). 

The measures of learning’s impact identified in this study demonstrate for practitioners that measures 
agreed upon by the organization as a whole are a blend of qualitative and quantitative measures of multiple 
operational and functional areas. Regardless of firm size, deconstructing business goals into questions such as “what 
does client satisfaction look like” or “what observed behaviors indicate that an employee is applying new learning or 
new knowledge” can yield an agreed upon set of measures to be used as a baseline against which to measure 
progress or to compare the current state of your learning impact and a target future state. 

Limitations 

Although our study provides insights into the measurement of the business impact of learning in 
professional services firm settings, the study does have limitations. One limitation of is the focus on one specific 
industry sector. Research has shown that the metrics and measures in the manufacturing sector (Lawler, Levenson, 

& Boudreau, 2004) or the retail sector (Ayad, 2008), for example, are different from those in the professional 
services sector. Another limitation is the small sample size (n=15), so that it is not feasible to generalize our findings 
to all professional services firms. Moreover, the firms represented in this study are large enterprises (median number 
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of employees=8,000). It is possible that small and medium-sized PSFs could have different metrics and methods for 
measuring the business impact of employee learning. Lastly, we operated under a particular conceptual lens which 
focused on individual learning and the impact it has on business performance. Consequently, that lens may have 
affected how the interview data was interpreted and that a different theoretical lens could lead to a different 
interpretation of the data. 

Future Research 

With a wide variety of factors influencing the way in which the business impact of learning is measured, 
we recommend that qualitative studies be an integral part of future research efforts. For example, replication of our 
study with participants from small and medium-sized professional services firms can be invaluable in understanding 
organizational, governance, and cultural variation both across and within professions that may affect how the 
business impact of employee learning is measured. Furthermore, it would be insightful to explore, at the individual 
level, how employees perceive the impact of their own learning activities and compare what employees designate as 
indicators of their contribution with those of their employers. This would assist FIR managers in identifying gaps 
between what the firm’s leadership values and what employees believe is (or should be) valued. Future research 
could also include case studies that flesh out the process of deciding what to measure, why the chosen measures are 
deemed to be important, and how those measures are communicated throughout the organization to promote a 
shared understanding of learning’s contribution to the business. 

Conclusion 

The results of our research suggests that the link between employee learning and the impact it has on a 
firm’s performance is supported through a variety of quantitative and qualitative measures, some of which relate to 
the firm’s success with clients, others to employee productivity and to the firm’s support for learning. Thus, learning 
executives and managers at various levels of an organization may want to include qualitative metrics that they may 
not have previously considered. We submit that there may be cogency to measuring the impact of employee 
learning through metrics that measure learning’s indirect contribution to business growth, provided that those 
metrics are agreed upon by the leadership and disseminated throughout the organization. 
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Abstract 

A growing body of research shows that deeper understanding of complex systems can be achieved by 
constructing causal diagrams to articulate and refine one’s understanding. This study sequentially analyzed students’ 
causal mapping behaviors to distinguish action sequences that students use to create diagrams of high versus low 
accuracy. The findings provide directions on how to standardize causal mapping software so that such tools can be 
used effectively to assess students’ understanding of complex systems. 

Introduction 

Causal diagrams can be used in science education as a tool to teach and assess learners’ understanding of 
complex phenomena. Given that causal maps in theory represent learner’s cognitive structures, complex reasoning, 
and conceptual development, causal diagrams have been used to elicit, articulate, refine, assess, and improve 
understanding of the causal mechanism underlying complex problems. In addition, maps can be used in particular to 
support collaborative learning when students compare their maps to identify, trigger, and focus group discussions 
around key differences in viewpoints. A number of studies have formulated metrics to measure the accuracy and 
structural attributes of students’ maps. Studies have been conducted to determine how different constraints imposed 
on the map construction process affect student’s maps and learning, as well as to develop software tools to automate 
and reliably measure both the accuracy and the structural attributes of maps. 

However, maps should not be used for large-scale assessments until students’ prior knowledge/skills with using 
maps are thoroughly examined. One way to address this problem is to identify and model the processes that lead 
students to produce more accurate causal diagrams. However, no prior research has empirically identified, tested, 
and validated specific action sequences despite the variety of existing guidelines on how to create causal maps, 
conduct root cause analysis and engage in systems thinking. As a result, this study addressed the following 
questions: 

1 . What patterns exist in the sequences of actions used by students that produce causal diagrams that are high 
versus low in accuracy? 

2. Which sequences of actions help students create more accurate causal diagrams? 

Method 

Procedures 

Seventeen graduate students in an online course on collaborative learning at a large south-eastern university 
were presented with an activity to formulate their personal theory in collaborative learning. Students were instructed 
to use jMAP (Figure 1) to individually create a causal diagram containing seven pre-defined variables believed to 
influence team effectiveness in collaborative learning groups. The goal was to construct a diagram to convey each 
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student’s personal theory as to how the six variables directly/indirectly affect team effectiveness. Students were 
presented a video and practice activity to learn how to move and re -position nodes, insert links between nodes and 
affected nodes, change the density of the link to convey relative level of impact (high, medium, low), and change 
direction of the causal relationship (positive = black, inverse = red). They logged into an Elluminate web- 
conferencing session to record the entire map construction session (limited to 45 minutes maximum). At the same 
time, the jMAP software logged each action students performed on their diagram from start to finish. 


Figure 1 . Instructor’s diagram used as criterion for assessing the accuracy of students’ diagrams 




Data Analysis 

The fourteen causal diagrams completed and submitted by students were imported into jMAP to assess each 
students’ diagram (Figure 2) in relation to the instructor’s diagram. The instructor’s diagram was based entirely on 
the findings from an empirical study that determined the relationships between the variables using path analysis. Six 
criterion measures/scores were each multiplied by 10 and summed to determine each students’ total score (Figure 3). 
The diagrams produced by students that achieved the top five and bottom five scores were selected for analysis. The 
jMAP software captured 26 types of mechanical actions students performed on their causal diagrams (Table 1). 
These 26 actions were collapsed and reduced to six codes to facilitate the process of identifying overall patterns in 
students’ actions and action sequences. The codes sequences were imported into the DAT software to: a) compute 
transitional probabilities between action pairs; b) compute z-scores at/? < .01 to identify behavior “patterns”; and c) 
generate transitional state diagrams to visually convey and compare behavioral patterns exhibited by students that 
achieved high versus low scores on their causal diagrams. 
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Figure 2 . jMAP’s visual and quantitative assessment of a student’s diagram 
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Figure 3 . Ranking of students based on causal diagram scores across six criterion measures 
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Table 1. Codes assigned to each mechanical actions students perform and recorded in the jMAP 
causal mapping software 


SuperCodes 

Codes 

Definitions 

- 

STRT 

reset the map and/or deleted old map 

LINK 

AODR 

added new link pointing to the right 


ADDL 

added new link pointing to the left 


ADDU 

added new link pointing up 


ADDD 

added new link pointing down 


LK2 

attached link to the affected node 

RELINK 

RLK1 

redirected the existing link to a new causal node 


RLK2 

redirected the existing link to a new affected node 

• 

ULK1 

detached the beginning tail of the link 

• 

ULK2 

detached the end of the link 

ATTR 

ATT- 

changed link to the color red to convey a negative or inverse relationship 


ATT+ 

changed link to the color black to convey a positive relationship 


ATT2L 

changed link to low level of impact 


ATT2M 

changed link to moderlate level of impact 


ATT2H 

changed link to high level of impact 

DEL 

DEL 

deleted the link 

MOVE 

MS 

moved a node (which was the same node as the last moved node) 


MDn 

moved node to the north of the previously moved node 


Mdne 

moved node to the NE of the previously moved node 


Mde 

moved node to the East of the previously moved node 


Mdse 

moved node to the SE of the previously moved node 


Mdse 

moved node to the South of the previously moved node 


MDsw 

moved node to the SW of the previously moved node 


MDw 

moved node to the West of the previously moved node 


MDnw 

moved node to the NW of the previously moved node 

COMM 

COM 

added a comment to the link to explain how the causal node affects the affected node 


CREV 

revised the existing comment on the given link 

- 

SAVE 

clicked the save button 


Note: The symbol -'identifies actions omitted from the sequential analysis 


Discussion of Main Findings 

Four behavioral patterns were exhibited by the low performers (Figure 4). The state diagram for high 
performers also revealed the same patterns. Flowever, they exhibited four additional patterns. These patterns help to 
explain the processes students can use produce more accurate diagrams. The high performers exhibited the tendency 
to: 1) specify the attributes of a causal link is inserted between two nodes; 2) add a new link immediately after they 
re-direct an existing link and 3) they deleted a link; and 4) delete another link immediately after deleting a link. 
These patterns serve as possible indicators to show when students may or may not be applying logic rules like the 
ones embedded within the REASON software - logic rules that can help students come to the important realization. 

The unique behavioral patterns observed among the high performers suggest that re-directing and deleting links 
are a critical part of the mapping process. To determine when Relink and Delete actions should be performed in 
relation to the most immediate preceding action, the historical state diagram (Figure 5) shows that: a) the event that 
was most likely to precede the Relink action was the Move action (62% of all preceding events) and Link action 
(100%) among the high and low performers, respectively; b) the event most likely to precede the Delete action was 
the Link action (45% of all preceding events) and Move action (37%) among the high and low performers, 
respectively. At the time of presentation, we will also present findings on the extent to which students used a 
backward/deductive vs. forward/inductive approach and depth vs. breadth-first approach and whether one particular 
approach produced more accurate causal maps. 
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Figure 4. Transitional state diagrams of action sequences performed by low vs. high performers 



Low Scorers 


High Scorers 


Notes: Black and gray arrows identify probabilities that are and are not significantly greater than expected. Arrows are weighted in direct proportion to the observed 
transitional probability. The first and second numerical value displayed in each node identifies the number of times the given action was performed and the number of 
events that followed the given action. The size of the glow emanating from each node conveys the number of times the given action was performed. 


Figure 5. Historical state diagrams of actions most/least likely to elicit target actions 




Low Scorers 


High Scorers 


Directions for future research 

The findings provide preliminary ideas as to what actions sequences should be facilitated to help students 
achieve more accurate and deeper understanding of complex systems. Some issues that require further research are: 
a) replicate this study with a larger sample; b) integrate the target action sequences directly into causal mapping 
software interface to conduct a controlled experimental test and determine if the target processes increases map 
accuracy regardless of students’ prior knowledge/understanding; c) examine to what extent the target processes are 
effective across different domains or topics that are or are not temporal in nature; and d) test other criterion measures 
for assessing accuracy. 
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